GH Repository · algorithmicsuperintelligence
optillm
Optimizing inference proxy for LLMs
- stars
- 4,263
- 30-day movement
- +21/day
- Related entries
- 60
- Connections
- 2
dockeragent-appprompt-engineeringchain-of-thoughtagentic-frameworkPythonoptimizationopenai-apillm-inferencemoaopenaiagentic-aiagentic-workflowproxy-serveragentspythonllmapi-gatewaymixture-of-expertsmonte-carlo-tree-searchgenailarge-language-modelsagentllmapi
optillm is an optimizing inference proxy for LLMs that sits between clients and OpenAI-compatible APIs, applying inference-time techniques to requests. Its topics indicate support for approaches like chain-of-thought, Monte Carlo tree search, mixture-of-agents, and mixture-of-experts, and it ships with a Dockerfile and Claude project context file.
You can point existing OpenAI-compatible clients at it to get optimization techniques applied to your LLM calls without changing your application code.
Use it to
- Proxy OpenAI-compatible LLM requests through inference optimization techniques
- Apply chain-of-thought or Monte Carlo tree search at inference time
- Experiment with mixture-of-agents and mixture-of-experts inference
- Route LLM traffic through a single API gateway
- Compare optimization strategies against a standard inference baseline
For Developers running LLM applications who want inference-time optimization
- Role
- agent-app
- Language
- Python
- Licence
- Apache-2.0
- Forks
- 383
- Open issues
- 19
- Last push
- 2026-07-18
- Latest release
- v0.0.1 · 2024-10-24
topicsllm-inferenceproxy-serveroptimizationchain-of-thoughtopenai-apimixture-of-experts