Anything not like your other LLM Stack with Custom Wrapper Layer & Multi-nodes – Part 4

Today, I’ll be sharing the fourth & final installment of the Colibri LLM framework with a bit of custom twists to get a stunning result at the end of this post.

In case, if you want read the previous posts from this series, then do visit the following links –

ANYTHING NOT LIKE YOUR OTHER LLM STACK – PART 1

ANYTHING NOT LIKE YOUR OTHER LLM STACK – PART 2

ANYTHING NOT LIKE YOUR OTHER LLM STACK – PART 3

The primary goal of this post is to optimize overall LLM response performance by finding the right balance between hardware capabilities and the optimization techniques offered by the Colibri framework. The objective is to achieve the best possible performance when running large language models (LLMs) locally, without relying on massive AI infrastructure or expensive cloud-based premium models.

But, before that let us watch the demo first.

Great! Based on the tests conducted across the use cases shown above, we observed significant differences in the total time required to generate a complete LLM response. The following summarizes the performance of each configuration using the same prompt and a response length of approximately 400 words:

  • Option 1 – Direct Execution Using the Colibri Framework (Command Line): Running the Colibri framework directly from the command line took between 16 and 22 minutes to generate the complete response.
  • Option 2 – Colibri Framework With a Wrapper Layer (Custom Web UI): Using a custom web interface with a wrapper layer built around the Colibri framework reduced the total response time to under 32 minutes for the same prompt.
  • Option 3 – Colibri Framework With a Wrapper Layer and Multi-Node Architecture (Custom Web UI):Integrating the wrapper layer with the Colibri framework in a multi-node configuration further reduced the total response time to 3 – 4 minutes or less, representing a substantial performance improvement over the previous configurations.

Key Takeaway: The multi-node configuration delivered the most significant performance improvement, reducing response generation time from over 16 to 22 minutes to 3 – 4 minutes or less. This demonstrates the potential of combining hardware resources with software optimization to make local LLM inference considerably faster, without relying on expensive cloud-based infrastructure.


Now, let’s explore how we achieved what initially seemed impossible. Before diving into the technical details, it is important to understand the core principles behind this solution. The key lies in a combination of software and hardware optimizations, along with efficient data transfer and communication between components. Each of these elements plays a critical role in achieving the performance improvements demonstrated above.

For this setup, I used a Satechi CubeDock Thunderbolt 5 docking station equipped with a Samsung 9100 PRO 2 TB NVMe SSD as an external high-speed storage solution, rather than upgrading the internal storage of my MacBook Pro M4 Max. This external SSD stores two different LLMs configured to run using the Colibri framework.

Since the MacBook Pro can directly access this high-speed SSD, I used SMB (Server Message Block) network sharing to make the model files accessible to both Mac minis. Each Mac mini mounts the shared storage over the network, allowing all three machines to access the centrally stored models without maintaining separate copies on their internal SSDs.

This approach helps overcome the individual storage limitations of the Mac minis while enabling a more flexible, multi-node architecture. However, shared storage does not increase the available RAM on each machine; memory utilization must still be managed according to each node’s hardware capabilities.

An important aspect of this architecture is that each LLM and computing node has a specific responsibility, while the MacBook Pro M4 Max serves as the primary or master node, coordinating the overall workflow and distributing tasks across the available resources.

Now, I’ll share the hardware details, which I bought it from Amazon & Quill (This is not a paid promotion for any of the products).

I’ve been using my Mac minis and MacBook Pro for quite some time, so I decided to put these existing machines to better use by combining them into a multi-node computing cluster.

You can adopt a similar approach using your existing computers, provided they meet the hardware and software requirements of your setup. The machines do not necessarily need identical specifications, as different nodes can contribute based on their individual capabilities.

The key takeaway? Don’t be too quick to discard your older computers! With the right configuration and optimization, they can still serve a valuable purpose in modern AI workloads. Unless you’re upgrading or trading them in as part of a worthwhile deal, consider how you might repurpose them. You never know when those additional computing resources might come in handy.


In this section, I’ll explain how each computing node can run a specific LLM based on its available memory and hardware capabilities, with each model assigned a particular task. By distributing these tasks across multiple nodes and executing them in parallel, we can significantly reduce the overall processing time and improve the efficiency of the entire system.

The goal is to demonstrate how this multi-node architecture can deliver competitive LLM response performance, potentially approaching that of commercial, cloud-based LLM services for certain workloads, while making the most of existing hardware resources and minimizing infrastructure costs.

As you can see that, the Mini’s will share the caches, whereas the Master node won’t as they belongs to complete two different kinds of model families.

Until or unless the Mac Mini’s are available the entire process will take place in the master node only.

Now, the sequence of activities that happened as follows ->

As soon as you submit the prompt the following sequence of activities will take place –

  • The master node (MacBook Pro M4 Max) first analyzes the submitted request and breaks it down into high-level sections based on the topic and its context.
    • Task Distribution: Once the sections are defined, the master node shares each section’s specific context and instructions with the assigned Mac mini node.
    • Parallel Processing: Each Mac mini processes one request at a time. With two Mac minis operating simultaneously, the system can handle up to two concurrent requests, allowing both nodes to independently generate relevant content for their assigned sections.
    • Task Queue and Monitoring: While both nodes are busy processing their current assignments, the remaining sections stay in a queue. The Web UI displays the status of each section, making it easy to identify which tasks are being processed and which are waiting for an available node.
    • Automatic Task Execution: As soon as a Mac mini completes its assigned section, it automatically picks up the next task from its queue. Using the context and instructions provided by the master node, it generates the content for that section. This process continues until all assigned sections are completed.
  • By distributing independent tasks across multiple computing nodes, this architecture enables parallel content generation, efficient resource utilization, and reduced overall response time compared with processing every section sequentially on a single node.
  • As soon as a child node (Mac mini) completes its assigned section, the master node (MacBook Pro M4 Max) begins the first phase of content review. During this phase, the master node checks for incomplete sentences or unfinished paragraphs. If any are identified, it analyzes the surrounding context and attempts to complete them while preserving the original meaning and flow.
    • Phase 1 – Section-Level Review: The master node reviews each completed section individually, identifying and correcting incomplete sentences or paragraphs to ensure the content is clear, coherent, and contextually relevant.
    • Phase 2 – Final Content Review and Refinement: Once all child nodes have completed their assigned sections and the first review phase is finished, the master node performs a second review of the entire response. During this phase, it identifies and removes duplicate statements, repetitive phrases, and redundant information across sections. It may also add important details or missing points that the child nodes did not include in their initial responses.
  • This two-phase review process helps improve the consistency, completeness, and overall quality of the final response while taking advantage of parallel content generation across multiple computing nodes.

The below image will explain the above-points clearly –

In the above, we’re showing the example of the following prompt -> “How to prepare Ilish-vape food in Bengali-Indian style?”

Hope this will give you clarity that you are looking for.

All the nodes will be connected by Thunderbolt 5 to get faster network communication.

And, here is the way all the nodes are connected ->

Satechi CubeDock (Samsung 2 TB SSD) -> MacBook Pro -> Mac Mini 1
-> Mac Mini 2

So, both the Mini’s are also diretly connected with the MacBook Pro as well.

Verify SMB is running on the MacBook Pro

networksetup -getinfo "Thunderbolt Bridge"

Verify the IP address & then execute the following –

sudo lsof -nP -iTCP:445 -sTCP:LISTEN

This checks whether the SMB service is listening for connections.

sudo launchctl print system/com.apple.smbd

Now, test the following in the Mini’s –

Configuring the SMB Drive:

nc -vz -G 5 192.168.100.X 445

X – represent your actual IP address (All the IP Address mentioned here are dummy in nature).

Then test the IP Address –

ping -c 20 192.168.100.Y

Remember, here the IP Address that is mentioned is the MacBook Pro’s IP Address i.e. Master node.

Mount the SSD from Finder

Using Command + K, you need to browse through the specific drive & navigate to the specific folder to mount it in the respective node.

smb://192.168.100.Y

Verify the SMB Drive from the terminal

mount | grep smbfs
ls /Volumes

It will show the newly mounted drive from the terminal as well.

The following is directory structure of the repo for the wrapper Layer that is hosted via the designed UI –

CollibriWrapper/
├── .cursor/
│   ├── hooks.json
│   ├── hooks/inbuilt-memory-capture.sh
│   └── rules/inbuilt-memory.mdc
├── .gitignore
├── .vscode/settings.json
├── config/nodes.json
├── docs/
│   ├── API.md
│   └── NODES_METHODS_AND_ALGORITHMS.md
├── backend/
│   ├── requirements.txt
│   ├── app/
│   │   ├── __init__.py
│   │   ├── capabilities.py
│   │   ├── colibri.py
│   │   ├── config.py
│   │   ├── discover.py
│   │   ├── hoststats.py
│   │   ├── main.py
│   │   ├── orchestrator.py
│   │   ├── planner.py
│   │   ├── scheduler.py
│   │   └── volumes.py
│   └── tests/
│       ├── __init__.py
│       ├── test_capabilities.py
│       ├── test_cluster_schedule.py
│       ├── test_discover.py
│       └── test_planner.py
├── frontend/
│   ├── index.html
│   ├── package.json
│   ├── package-lock.json
│   ├── tsconfig.json
│   ├── vite.config.ts
│   └── src/
│       ├── App.tsx
│       ├── main.tsx
│       ├── styles.css
│       └── types.ts
├── scripts/
│   ├── colibri-volume.sh
│   ├── lib/libomp.dylib
│   ├── restart-from-ui.sh
│   ├── stack-common.sh
│   ├── start.sh
│   ├── start-wrapper.sh
│   ├── stop.sh
│   ├── worker-host.py
│   ├── worker-start.sh
│   └── worker-stop.sh
└── data/
    ├── run/
    │   ├── canvas.log
    │   ├── colibri.log
    │   ├── colibri.volume
    │   ├── memory.log
    │   ├── memory-node.pid
    │   └── wrapper.log
    └── sessions/
        ├── 114af16ca1514055b309408288d8432a.json
        └── 686e1efe04f94ca293aac47e0f75ba5c.json

Start The Master Node:

We’ll be browsing through the right path (In our case) ->

cd /Volumes/TD_BLACK/StartUp/CollibriWrapper/scripts

# Start the Master Node
./start.sh

Stop The Master Node:

# To Stop the Master Node
./stop.sh

Start the Worker Node:

./worker-start.sh

Stop the Worker Node:

./worker-stop.sh

So, we’ve done it. 🙂

I hope you all like this effort & let me know your feedback. I’ll be back with another topic. Until then, Happy Avenging!

Anything not like your other LLM Stack – Part 2

In the part – 1, we’ve discussed the details on the basic concept of this enw LLM frame work & how it enables the similar capability without the need for accelerated GPUs or specific hardwares. In this post, we’ll further discuss this & explore more details on top of that.


The engine records routing activity and can keep frequently used experts in faster memory. It combines mechanisms such as:

NVMe → RAM caching → pinned hot experts → optional GPU residency → prefetching

The project describes this as analogous to a JIT compiler for weights: instead of compiling frequently executed code, it learns which model experts are frequently requested and moves those experts into faster storage tiers. 

This is why repeated workloads can behave differently from completely cold inference.


Colibrì can also distribute expert reads across two storage devices.

Because disk bandwidth is frequently the limiting factor, the engine supports a second model mirror and routes reads across the drives according to measured or configured bandwidth.

And, we’ll be also exploring this appraoch to reduce the load in our MacBook Pro.

So something conceptually like:

                     ┌── NVMe SSD #1
Model expert request ┤
└── NVMe SSD #2
↓
RAM
↓
GPU

This can increase the effective streaming bandwidth.

That is a fairly distinctive capability compared with conventional local LLM runtimes.

The API also supports persistent KV contexts.

Colibrì can save KV state to .coli_kv files and reuse the common prefix when a conversation continues. For more information please refer the following link.

That can matter enormously for huge models because repeatedly processing a long conversation history could otherwise be extremely expensive.

Multiple isolated KV slots are also supported, currently up to 16 according to the API documentation. Refer the below link to know more on this.

Colibrì demonstrates that enormous models can run on modest hardware.

That does not mean they run at cloud-API speeds. The project’s own benchmark documentation is unusually transparent about this.

For the 744B GLM model, documented examples include roughly:

  • 25 GB development machine, cold: 0.05–0.1 tokens/s
  • 128 GB CPU-only system, warm: ~1.8 tokens/s
  • RTX 5070 Ti system: ~1.07 tokens/s in one documented configuration
  • Large six-RTX-5090 configuration: several tokens per second with experts resident rather than coming from disk.

In Colibri, the coding agent might send 10,000–20,000 tokens before the user’s first message, and disk-streamed prefill could therefore take an extremely long time.

Colibrì’s innovation is primarily accessibility and heterogeneous resource utilization, not magically eliminating the physics of memory bandwidth.

The key contribution is not simply “running an LLM locally.” Tools such as Ollama already make that common.

Colibrì is exploring a different question:

What if model size no longer had to be constrained by available RAM or VRAM?

The repository treats storage, memory, caching, routing patterns, speculative decoding, quantization, NUMA, CPU/GPU overlap, and expert placement as parts of the inference system itself.

That could make it valuable for four broad areas:

  • Local/private inference, particularly when sending data to an external inference provider is undesirable.
  • AI systems research, especially research into MoE routing, caching, memory hierarchy, quantization, and heterogeneous computing.
  • Low-cost experimentation with frontier-scale open models, where owning several data-center GPUs would otherwise be required.
  • Building local AI services, because its OpenAI/Anthropic-compatible APIs can sit underneath applications, RAG systems, coding tools, or agent frameworks.

One qualification is important: local execution can improve control over where prompts and model execution occur, but “local” by itself should not be interpreted as an automatic security or privacy guarantee. Network exposure, APIs, applications connected to the server, file access, and logging still need to be configured appropriately. Colibrì defaults the server to localhost and recommends setting an API key before exposing it beyond the machine.


Now, let us understand how we want to proceed with our architecture implementation locally even further reducing the need for the SSD memories of my Macbook Pro.

Colibrì supports a useful feature called a partial model mirror. You can retain the complete model on your external drive like mine (SD_BLACK) and copy a selected portion of frequently used model shards onto your Mac’s much faster internal SSD. Colibrì then reads from both locations. Its official documentation provides commands to plan, stage and verify these partial mirrors. I tried both the options, partially, stored in Macbook & partially stored in my high speed SSD USB. And, alo test by placing the complete model in my MacBook Pro to understand the implications of the performance for storing them in different places.

Now, what my external drive will tell us about the performance –

I’ve used a faster SSD USB from SanDisk to get the better response, while storing the Colibri library over there. That way I can get the almost similar response from the external SSD USB like my internal Macbook Pro’s SSD.

SanDisk officially specifies the Extreme Portable SSD at up to 1,050 MB/s sequential read and 1,000 MB/s sequential write, using USB 3.2 Gen 2. For more on this device, you can refer the following link.

Let us understand, how the cache works for us –

From thje above image, you can see that by default, the main models stays in the faster SanDisk SSD USB. However, 100 GB of the mirror models are kept inside the main laptop’s SSD for even better performance. So, it is a mid-level tradeoff againsts the lapotp’s memory utilization Vs performance.

Using the following way, you can configure the model endpoint ->

export COLI_MODEL="/Volumes/SD_BLACK/AI/models/glm52_i4"
export MIRROR="$HOME/ColibriMirror/glm52_i4"

df -h "$HOME"
mkdir -p "$MIRROR"

cd /Volumes/SD_BLACK/AI/colibri/c

./coli mirror plan \
  --model "$COLI_MODEL" \
  --mirror "$MIRROR" \
  --budget-gib 100 \
  --reserve-gib 60

Review the plan first. Provided you have sufficient internal SSD space, stage and verify the selected shards:

./coli mirror stage \
  --model "$COLI_MODEL" \
  --mirror "$MIRROR" \
  --budget-gib 100 \
  --reserve-gib 60

./coli mirror verify \
  --model "$COLI_MODEL" \
  --mirror "$MIRROR"

These commands are supported by the current repository; an older local Colibrì executable may need updating if it does not recognize coli mirror. Staging preserves the original external model and verifies copied shards. For more information on this, please refer to the following link.

After successful verification, start the server with the mirror enabled:

COLI_MODEL_MIRROR="$MIRROR" \
COLI_METAL=1 \
CAP_RAISE=0 \
MTP=0 \
./coli serve \
  --model "$COLI_MODEL" \
  --ram 88 \
  --cap 8 \
  --host 127.0.0.1 \
  --port 8000 \
  --model-id glm-5.2-colibri

Now, in the next thread we’ll understand the entire step-by-step setup of this initiative & explian one more complex combinations to get even better resolution with custom twists.

Till then, Happy Avenging! 😀

Anything not like your other LLM Stack – Part 1

Hi Guys,

Today, I’ll be presenting a brand new LLM that you can configure locally & based on your computation, I/O & other factors. We’re going to understand the open-source project named -> Colibri.

We’ll be presenting a series of simple & complex process, which will be partially optimized based on various scenarious.

Colibrì is an open-source inference engine for running very large Mixture-of-Experts (MoE) AI models locally. Its central idea is unusual: instead of requiring hundreds of gigabytes—or even terabytes—of model weights to fit entirely in RAM or GPU VRAM, it treats VRAM + RAM + NVMe storage as one memory hierarchy.

The router determines which experts are needed for each token, and Colibrì loads or caches those experts as required. In other words, it effectively performs just-in-time loading of model weights.


Let us understand the overall capability in a simpler form –

CapabilityWhat Colibrì provides
Local large-model inferenceRuns very large MoE models locally rather than requiring a hosted API.
Models larger than RAM/VRAMStreams routed experts from SSD/NVMe instead of loading the full model into memory.
CPU-only operationA GPU is not mandatory. CPU-only operation is supported.
GPU accelerationSupports heterogeneous acceleration paths including CUDA, Metal, Vulkan, and AMD/HIP in documented configurations.
Memory multitieringModel weights can be distributed across VRAM → RAM → NVMe according to available hardware.
Large open modelsCurrently documents support for nine MoE model families, including GLM, Kimi, DeepSeek, Qwen, Inkling, and OLMoE.
Local Chat interfacecoli chat provides direct interactive inference.
Web UIcoli web provides a browser-based interface with performance information and model visualization.
OpenAI-compatible APIcoli serve exposes endpoints such as /v1/chat/completions and /v1/models.
Anthropic-compatible APIAlso provides /v1/messages, allowing some Anthropic-compatible software to communicate with Colibrì.
Tool/function callingAvailable for certain supported engines, including GLM, DeepSeek V4, and Kimi K3.
Coding-tool integrationCan act as a local backend for compatible coding applications that accept an OpenAI-compatible endpoint.
Persistent KV cacheConversation state can be persisted and reused rather than completely prefilling the model again.
Multi-machine experimentsIncludes an experimental/local cluster mechanism in which other Macs can execute routed expert FFNs.
Research/observabilityProvides expert routing telemetry, profiling, Brain visualization, Atlas visualization, benchmarking, and experimentation facilities.
Open sourceEngine is released under Apache 2.0.

For more information, please refer to the following link.

Let us understand how this works for the larger models –

For the project’s reference GLM-5.2 model, the approximately 430 GB quantized model does not need 430 GB of RAM. The documentation describes keeping roughly 9.9 GB of dense components resident while streaming routed experts from storage as needed.

Please go throught he following link for more details.

Let us calculate it –

Model → must largely fit RAM/VRAM → inference
NVMe ↔ RAM ↔ VRAM → dynamically selected experts → inference

Colibrì calls this a multitier hierarchy. Fast memory affects performance, but the design aims to avoid changing the model merely because less VRAM/RAM is available. For more information, let us understand it from the official link.

That architectural concept is probably the project’s most significant contribution.

Model familyApprox. total parametersApprox. storage/RAM characteristics documented by Colibrì
GLM-5.2 / GLM-5.3744B~372 GB disk; 16 GB minimum RAM
GLM-5.3-Flash321B~195 GB converted; ~25 GB RAM
Inkling975B~469 GB; ~25 GB RAM with int4 dense container
Kimi K32.8T~1.6 TB disk; 32 GB+ RAM
DeepSeek V4 Flash284B~167 GB; 16 GB minimum
DeepSeek V4.1 Flash552Bseparate current engine implementation
Qwen3.8-Flash-Next125B + 51B n-gram~185.5 GB; 16 GB minimum
Qwen3.6-35B-A3B35B / ~3B active~20 GB; ~24 GB RAM
OLMoE7B~7 GB; ~8 GB RAM

The current repository specifically describes Kimi K3 at 2.8 trillion parameters, which illustrates how far the storage-streaming approach is intended to scale. One important distinction: Colibrì does not create these models. It is the inference engine that runs compatible open-weight models.

This is another major difference from frameworks primarily designed around GPU residency. The official quick-start says the baseline requirements for GLM-5.2 are approximately:

~16 GB RAM minimum, ~380 GB free storage, with fast NVMe recommended.

A GPU is optional. Refer to this link.

A GPU can significantly accelerate inference, but Colibrì’s architecture intentionally allows CPU + RAM + SSD operation. The repository currently contains backends for CUDA, Metal, Vulkan, and AMD/HIP-related configurations. Please refer the following link.

This makes Colibrì particularly interesting for heterogeneous hardware rather than only expensive data-center GPU configurations.

So, you don’t need the GPU to run any model.


This could be one of its most useful practical capabilities.

Run the following command:

coli serve

Creates an OpenAI-compatible HTTP API. The official documentation currently lists endpoints, including:

GET /v1/models

GET /v1/models/{model}

POST /v1/chat/completions

POST /v1/completions

It supports streaming responses, token usage, temperature, top-p, maximum-token parameters, and stop sequences.

So, you can use the application either from the postman or using API-based call or from using a curl statement while invoking the following entries –

http://localhost:8000/v1

That makes Colibrì much more than a command-line model runner.

Interestingly, Colibrì now exposes:

/v1/messages

for software built around the Anthropic Messages API.

The documentation specifically discusses using this interface with compatible clients, including Claude Code-style integrations. For more information, please refer the following link.

Therefore, the architecture can look approximately like:

This opens some interesting possibilities for private local AI applications, coding assistants, RAG systems, research environments, and experimental agents.


Some of the models support tool/function calling. This is particularly relevent if you are thinking of agentic systems. So, Colibri potentially provides the LLM reasoing/tool-call layer of a local agent architecture.

Colibrì itself isn’t a complete agent orchestration framework like AutoGen or similar systems; it provides the model inference/API layer on which an agent framework could operate.

Colibri also offered web dashboard accessible through:

coli web

The project describes three particularly interesting views –

Dashboard provides token-generation and performance metrics, including time breakdowns and VRAM/RAM/disk utilization.

Brain visualizes the model’s experts and shows which experts activate during generation.

Atlas visualizes measured expert specialization and routing affinities as a three-dimensional representation.

That makes Colibrì interesting not only for inference but also for studying how MoE models route information internally.

For more information, you can refer the following link.


Let us continue this topic in our next series.

Until then, Happy Avenging! 🙂