Saltar a contenido

apogee-ai-serving

Tier-aware router (CHEAP/BALANCED/EXPENSIVE), shadow traffic, canaries, batch inference.

This page is being expanded

The tables below are generated from the package source, so they are accurate. Narrative guides and worked examples are being written module by module.

Public API

Everything below is exported from the package root, so from apogee_ai_serving import X works for each name.

Layer Symbols
Application · DTOs BenchDTO, CompleteDTO, RouteDTO
Application · Use cases BenchCacheHitUseCase, CompleteUseCase, ListEnginesUseCase, RouteUseCase
Domain CacheKey, CacheLookup, FallbackPolicy, ModelDescriptor, ModelTier, QuotaScope, QuotaSnapshot, RateLimit, RouterRule, RoutingDecision, RoutingStrategy, ServingChunk, ServingCompletionRequest, ServingCompletionResponse
Domain · Enums CacheKind, EngineKind
Domain · Exceptions EngineNotAvailableException, FallbackExhaustedException, ModelNotSupportedException, NoEngineMatchedException, RateLimitExceededException, ServingError
Domain · Protocols (ports) ICompletionEngine, IFallbackChain, IModelRouter, IPromptEmbedder, IRateLimiter, ISemanticCache
Infrastructure BridgedPromptEmbedder, EchoEngine, EngineRegistry, HashingPromptEmbedder, InMemoryRateLimiter, InMemorySemanticCache, JsonSemanticCache, LMStudioEngine, LlamaCppEngine, MlxEngine, OllamaEngine, OrderedFallbackChain, RedisRateLimiter, RedisSemanticCache, TgiEngine, TierAwareRouter, VllmEngine

Install

Bash
pip install apogee-ai-serving

The base install has no required third-party dependency. Each extra unlocks one adapter.

Extra Pulls in
ollama ollama>=0.3
vllm vllm>=0.6
llamacpp llama-cpp-python>=0.2
mlx mlx-lm>=0.18
redis redis>=5.0
embeddings apogee-ai-providers>=0.1
all ollama>=0.3, vllm>=0.6, llama-cpp-python>=0.2, mlx-lm>=0.18, redis>=5.0, apogee-ai-providers>=0.1

CLI

Installed as a console script, and registered as an apogee plugin so the same subcommands work under the global binary.

Bash
apogee-serving --help