Anything not like your other LLM Stack – Part 1

Hi Guys,

Today, I’ll be presenting a brand new LLM that you can configure locally & based on your computation, I/O & other factors. We’re going to understand the open-source project named -> Colibri.

We’ll be presenting a series of simple & complex process, which will be partially optimized based on various scenarious.

Colibrì is an open-source inference engine for running very large Mixture-of-Experts (MoE) AI models locally. Its central idea is unusual: instead of requiring hundreds of gigabytes—or even terabytes—of model weights to fit entirely in RAM or GPU VRAM, it treats VRAM + RAM + NVMe storage as one memory hierarchy.

The router determines which experts are needed for each token, and Colibrì loads or caches those experts as required. In other words, it effectively performs just-in-time loading of model weights.

Let us understand the overall capability in a simpler form –

For more information, please refer to the following link.

Why large models works here?

Let us understand how this works for the larger models –

For the project’s reference GLM-5.2 model, the approximately 430 GB quantized model does not need 430 GB of RAM. The documentation describes keeping roughly 9.9 GB of dense components resident while streaming routed experts from storage as needed.

Please go throught he following link for more details.

Let us calculate it –

Traditional local inference:

Model → must largely fit RAM/VRAM → inference

Colibrì:

NVMe ↔ RAM ↔ VRAM → dynamically selected experts → inference

Colibrì calls this a multitier hierarchy. Fast memory affects performance, but the design aims to avoid changing the model merely because less VRAM/RAM is available. For more information, let us understand it from the official link.

That architectural concept is probably the project’s most significant contribution.

Supported Models:

The current repository specifically describes Kimi K3 at 2.8 trillion parameters, which illustrates how far the storage-streaming approach is intended to scale. One important distinction: Colibrì does not create these models. It is the inference engine that runs compatible open-weight models.

Do you need GPU to run this?

This is another major difference from frameworks primarily designed around GPU residency. The official quick-start says the baseline requirements for GLM-5.2 are approximately:

~16 GB RAM minimum, ~380 GB free storage, with fast NVMe recommended.

A GPU is optional. Refer to this link.

A GPU can significantly accelerate inference, but Colibrì’s architecture intentionally allows CPU + RAM + SSD operation. The repository currently contains backends for CUDA, Metal, Vulkan, and AMD/HIP-related configurations. Please refer the following link.

This makes Colibrì particularly interesting for heterogeneous hardware rather than only expensive data-center GPU configurations.

So, you don’t need the GPU to run any model.

Use it like a OpenAI Server:

This could be one of its most useful practical capabilities.

Run the following command:

coli serve

coli serve

Creates an OpenAI-compatible HTTP API. The official documentation currently lists endpoints, including:

GET /v1/models

GET /v1/models/{model}

POST /v1/chat/completions

POST /v1/completions

GET /v1/models

GET /v1/models/{model}

POST /v1/chat/completions

POST /v1/completions

It supports streaming responses, token usage, temperature, top-p, maximum-token parameters, and stop sequences.

So, you can use the application either from the postman or using API-based call or from using a curl statement while invoking the following entries –

http://localhost:8000/v1

http://localhost:8000/v1

That makes Colibrì much more than a command-line model runner.

Use it like a Anthropic Server:

Interestingly, Colibrì now exposes:

/v1/messages

/v1/messages

for software built around the Anthropic Messages API.

The documentation specifically discusses using this interface with compatible clients, including Claude Code-style integrations. For more information, please refer the following link.

Therefore, the architecture can look approximately like:

This opens some interesting possibilities for private local AI applications, coding assistants, RAG systems, research environments, and experimental agents.

Some of the models support tool/function calling. This is particularly relevent if you are thinking of agentic systems. So, Colibri potentially provides the LLM reasoing/tool-call layer of a local agent architecture.

Colibrì itself isn’t a complete agent orchestration framework like AutoGen or similar systems; it provides the model inference/API layer on which an agent framework could operate.

Colibri also offered web dashboard accessible through:

coli web

coli web

The project describes three particularly interesting views –

Dashboard provides token-generation and performance metrics, including time breakdowns and VRAM/RAM/disk utilization.

Brain visualizes the model’s experts and shows which experts activate during generation.

Atlas visualizes measured expert specialization and routing affinities as a three-dimensional representation.

That makes Colibrì interesting not only for inference but also for studying how MoE models route information internally.

For more information, you can refer the following link.

Let us continue this topic in our next series.

Until then, Happy Avenging! 🙂

Note: All the data & scenarios posted here are representative of data & scenarios available on the internet for educational purposes only. There is always room for improvement in this kind of model & the solution associated with it. This article is for educational purposes only. The techniques described should only be used for authorized security testing and research. Unauthorized access to computer systems is illegal and unethical & not encouraged.