AI agent tools for LLM inference
Short answer: Our top pick for LLM inference: lemonade, with the highest AgentPick Score (80/100) among tools tagged "llm-inference". Runners-up: neuron-ai (74) and MTPLX (73). Compare what each does, its license and how recently it was updated below.
| # | Tool | What it does | Score | Stars | License | Updated |
|---|---|---|---|---|---|---|
| #1 | lemonade | Lemonade helps users discover and run local AI apps by serving optimized LLMs right from their own GPUs and NPUs. Join our discord: https://discord.gg… | 80/100 | ★ 5.9k | Apache-2.0 | 14 Oct 2026 |
| #2 | neuron-ai | The Agentic Framework of the PHP ecosystem to build production-ready AI driven applications. Connect components (LLMs, Tools, vector DBs, memory) to a… | 74/100 | ★ 2.1k | MIT | 14 Oct 2026 |
| #3 | MTPLX | The fastest way to run Qwen 3.8 Flash Next, Qwen 3.8 27B and Ternary Bonsai 2 27B on a Mac: 125 tok/s in OpenCode on an M5 Max, and a 27B model on 16 … | 73/100 | ★ 2.6k | Apache-2.0 | 10 Oct 2026 |
| #4 | Rapid-MLX | Rapid-MLX is an open-source (Apache 2.0) OpenAI- and Anthropic-compatible LLM inference server for Apple Silicon, built on MLX, focused on reliable to… | 72/100 | ★ 4k | See repo | 14 Oct 2026 |
| #5 | optillm | Optimizing inference proxy for LLMs | 71/100 | ★ 4.3k | Apache-2.0 | 11 Oct 2026 |
| #6 | Atomic-Chat | Local AI app and inference engine for agents. Run open-weight LLMs locally — private, 100% offline on your computer. Join our Discord: https://discord… | 70/100 | ★ 1.8k | See repo | 14 Oct 2026 |
| #7 | nanocoder | An open coding agent for your terminal, built by a community collective rather than a company. Bring your own model, keep your code on your machine, a… | 69/100 | ★ 2.5k | See repo | 14 Oct 2026 |
| #8 | InferenceX | Open Source AI Accelerator Research Platform Standard / 开源推理研究平台 | 68/100 | ★ 1.8k | Apache-2.0 | 14 Oct 2026 |
| #9 | taOS | Self-hosted AI agent OS. Your memory, chat, agents, and files stay on hardware you own, offline by default, cloud by choice. Offline AI memory (taOSmd… | 61/100 | ★ 561 | AGPL-3.0 | 13 Oct 2026 |
| #10 | mlx-dspark | Up to 4× faster LLM decoding on Apple Silicon, lossless. Native MLX port of DeepSeek's DSpark & z-lab's DFlash speculative decoding — Gemma-4, Qwen3.8… | 60/100 | ★ 715 | MIT | 5 Oct 2026 |
| #11 | edsl | Design, conduct and analyze results of AI-powered surveys and experiments. Simulate social science and market research with large numbers of AI agents… | 44/100 | ★ 504 | MIT | 12 Oct 2026 |
| #12 | Context-Engine | Context-Engine MCP - Agentic Context Compression Suite | 35/100 | ★ 401 | MIT | 8 Jul 2026 |
| #13 | kaggle-tpu-lab | Frontier-class open models on a free Kaggle TPU v5e-8: GLM-5.3-Flash 320B MoE (~165 tok/s, our own JAX engine) and Qwen3.8-27B bf16 (~130 tok/s), 262k… | 32/100 | ★ 637 | MIT | 6 Oct 2026 |
| #14 | KVarN | KVarN is a native vLLM KV-cache quantization backend for your agents: 3-5x more context, throughput above FP16, and FP16-level accuracy. Calibration-f… | 26/100 | ★ 509 | Apache-2.0 | 22 Jun 2026 |
Source: each project's public GitHub repository (description, stars, license, last push). Listed tools aren't tested or ranked by AgentPick.
How to choose
- Read the license before using it commercially.
- Check how recently it was updated and how active its issues are.
- Run it on a small, low-risk task first.
Get the best new agent tools every week. Free, with a verdict on each. Get the weekly digest
Related
AI agent tools for agent memory →
Agent building
AI agent tools for self-hosted setups →
Agent building
AI agent tools for LangChain →
Agent building
AI agent tools for sandbox →
Agent building
AI agent tools for harness engineering →
Agent building
AI agent tools for penetration testing →
Agent building
AgentPick Weekly
The week's new agent tools with a verdict on each, every week. Free. We'll email you a link to confirm, and you can unsubscribe any time. See our Privacy Policy.