​
Overview

Atoma Node integrates several leading open-source inference engines:

  • vLLM: A high-throughput and memory-efficient inference engine optimized for LLMs. Features state-of-the-art serving throughput with PagedAttention memory management and continuous batching.

  • mistral.rs: A blazingly fast Rust-based inference engine with support for various model architectures, quantization methods, and hardware acceleration options.

  • Text Embeddings Inference (TEI): A high-performance solution specifically designed for text embedding models, offering both REST and gRPC APIs with support for various embedding model architectures.

​
Chat Completions

BackendArchitecture/PlatformDocker Compose Profile
vLLMCUDAchat_completions_vllm
vLLMx86_64chat_completions_vllm_cpu
vLLMROCmchat_completions_vllm_rocm
mistral.rsx86_64, aarch64chat_completions_mistralrs_cpu

​
Embeddings

BackendArchitecture/PlatformDocker Compose Profile
Text Embeddings InferenceCUDAembeddings_tei

​
Image Generations

BackendArchitecture/PlatformDocker Compose Profile
mistral.rsCUDAimage_generations_mistralrs

To run the node with confidential compute mode, you can use the following command:

# Build and start all services
COMPOSE_PROFILES=chat_completions_vllm,embeddings_tei,image_generations_mistralrs,confidential docker compose up --build

# Only start one service
COMPOSE_PROFILES=chat_completions_vllm,confidential docker compose up --build

# Run in detached mode
COMPOSE_PROFILES=chat_completions_vllm,embeddings_tei,image_generations_mistralrs,confidential docker compose up -d --build

Otherwise, you can run the node in non-confidential mode with:

# Build and start all services
COMPOSE_PROFILES=chat_completions_vllm,embeddings_tei,image_generations_mistralrs,non-confidential docker compose up --build

# Only start one service
COMPOSE_PROFILES=chat_completions_vllm,non-confidential docker compose up --build

# Run in detached mode
COMPOSE_PROFILES=chat_completions_vllm,embeddings_tei,image_generations_mistralrs,non-confidential docker compose up -d --build
Overview