This site mainly deals with various use cases demonstrated using Python, Data Science, Cloud basics, SQL Server, Oracle, Teradata along with SQL & their implementation. Expecting yours active participation & time. This blog can be access from your TP, Tablet & mobile also. Please provide your feedback.
Today, I’ll be sharing the fourth & final installment of the Colibri LLM framework with a bit of custom twists to get a stunning result at the end of this post.
In case, if you want read the previous posts from this series, then do visit the following links –
The primary goal of this post is to optimize overall LLM response performance by finding the right balance between hardware capabilities and the optimization techniques offered by the Colibri framework. The objective is to achieve the best possible performance when running large language models (LLMs) locally, without relying on massive AI infrastructure or expensive cloud-based premium models.
But, before that let us watch the demo first.
Great! Based on the tests conducted across the use cases shown above, we observed significant differences in the total time required to generate a complete LLM response. The following summarizes the performance of each configuration using the same prompt and a response length of approximately 400 words:
Option 1 – Direct Execution Using the Colibri Framework (Command Line): Running the Colibri framework directly from the command line took between 16 and 22 minutes to generate the complete response.
Option 2 – Colibri Framework With a Wrapper Layer (Custom Web UI): Using a custom web interface with a wrapper layer built around the Colibri framework reduced the total response time to under 32 minutes for the same prompt.
Option 3 – Colibri Framework With a Wrapper Layer and Multi-Node Architecture (Custom Web UI):Integrating the wrapper layer with the Colibri framework in a multi-node configuration further reduced the total response time to 3 – 4 minutes or less, representing a substantial performance improvement over the previous configurations.
Key Takeaway: The multi-node configuration delivered the most significant performance improvement, reducing response generation time from over 16 to 22 minutes to 3 – 4 minutes or less. This demonstrates the potential of combining hardware resources with software optimization to make local LLM inference considerably faster, without relying on expensive cloud-based infrastructure.
Now, let’s explore how we achieved what initially seemed impossible. Before diving into the technical details, it is important to understand the core principles behind this solution. The key lies in a combination of software and hardware optimizations, along with efficient data transfer and communication between components. Each of these elements plays a critical role in achieving the performance improvements demonstrated above.
ARCHITECTURE:
For this setup, I used a Satechi CubeDock Thunderbolt 5 docking station equipped with a Samsung 9100 PRO 2 TB NVMe SSD as an external high-speed storage solution, rather than upgrading the internal storage of my MacBook Pro M4 Max. This external SSD stores two different LLMs configured to run using the Colibri framework.
Since the MacBook Pro can directly access this high-speed SSD, I used SMB (Server Message Block) network sharing to make the model files accessible to both Mac minis. Each Mac mini mounts the shared storage over the network, allowing all three machines to access the centrally stored models without maintaining separate copies on their internal SSDs.
This approach helps overcome the individual storage limitations of the Mac minis while enabling a more flexible, multi-node architecture. However, shared storage does not increase the available RAM on each machine; memory utilization must still be managed according to each node’s hardware capabilities.
An important aspect of this architecture is that each LLM and computing node has a specific responsibility, while the MacBook Pro M4 Max serves as the primary or master node, coordinating the overall workflow and distributing tasks across the available resources.
Now, I’ll share the hardware details, which I bought it from Amazon & Quill (This is not a paid promotion for any of the products).
I’ve been using my Mac minis and MacBook Pro for quite some time, so I decided to put these existing machines to better use by combining them into a multi-node computing cluster.
You can adopt a similar approach using your existing computers, provided they meet the hardware and software requirements of your setup. The machines do not necessarily need identical specifications, as different nodes can contribute based on their individual capabilities.
The key takeaway? Don’t be too quick to discard your older computers! With the right configuration and optimization, they can still serve a valuable purpose in modern AI workloads. Unless you’re upgrading or trading them in as part of a worthwhile deal, consider how you might repurpose them. You never know when those additional computing resources might come in handy.
MODEL-DRIVEN ARCHITECTURE:
In this section, I’ll explain how each computing node can run a specific LLM based on its available memory and hardware capabilities, with each model assigned a particular task. By distributing these tasks across multiple nodes and executing them in parallel, we can significantly reduce the overall processing time and improve the efficiency of the entire system.
The goal is to demonstrate how this multi-node architecture can deliver competitive LLM response performance, potentially approaching that of commercial, cloud-based LLM services for certain workloads, while making the most of existing hardware resources and minimizing infrastructure costs.
As you can see that, the Mini’s will share the caches, whereas the Master node won’t as they belongs to complete two different kinds of model families.
Until or unless the Mac Mini’s are available the entire process will take place in the master node only.
Now, the sequence of activities that happened as follows ->
As soon as you submit the prompt the following sequence of activities will take place –
The master node (MacBook Pro M4 Max) first analyzes the submitted request and breaks it down into high-level sections based on the topic and its context.
Task Distribution: Once the sections are defined, the master node shares each section’s specific context and instructions with the assigned Mac mini node.
Parallel Processing: Each Mac mini processes one request at a time. With two Mac minis operating simultaneously, the system can handle up to two concurrent requests, allowing both nodes to independently generate relevant content for their assigned sections.
Task Queue and Monitoring: While both nodes are busy processing their current assignments, the remaining sections stay in a queue. The Web UI displays the status of each section, making it easy to identify which tasks are being processed and which are waiting for an available node.
Automatic Task Execution: As soon as a Mac mini completes its assigned section, it automatically picks up the next task from its queue. Using the context and instructions provided by the master node, it generates the content for that section. This process continues until all assigned sections are completed.
By distributing independent tasks across multiple computing nodes, this architecture enables parallel content generation, efficient resource utilization, and reduced overall response time compared with processing every section sequentially on a single node.
As soon as a child node (Mac mini) completes its assigned section, the master node (MacBook Pro M4 Max) begins the first phase of content review. During this phase, the master node checks for incomplete sentences or unfinished paragraphs. If any are identified, it analyzes the surrounding context and attempts to complete them while preserving the original meaning and flow.
Phase 1 – Section-Level Review: The master node reviews each completed section individually, identifying and correcting incomplete sentences or paragraphs to ensure the content is clear, coherent, and contextually relevant.
Phase 2 – Final Content Review and Refinement: Once all child nodes have completed their assigned sections and the first review phase is finished, the master node performs a second review of the entire response. During this phase, it identifies and removes duplicate statements, repetitive phrases, and redundant information across sections. It may also add important details or missing points that the child nodes did not include in their initial responses.
This two-phase review process helps improve the consistency, completeness, and overall quality of the final response while taking advantage of parallel content generation across multiple computing nodes.
The below image will explain the above-points clearly –
In the above, we’re showing the example of the following prompt -> “How to prepare Ilish-vape food in Bengali-Indian style?”
Hope this will give you clarity that you are looking for.
SETTING-UP YOUR INFRA:
All the nodes will be connected by Thunderbolt 5 to get faster network communication.
And, here is the way all the nodes are connected ->
Satechi CubeDock (Samsung 2 TB SSD) -> MacBook Pro -> Mac Mini 1
-> Mac Mini 2
So, both the Mini’s are also diretly connected with the MacBook Pro as well.
Verify SMB is running on the MacBook Pro
networksetup-getinfo"Thunderbolt Bridge"
Verify the IP address & then execute the following –
sudolsof-nP-iTCP:445-sTCP:LISTEN
This checks whether the SMB service is listening for connections.
sudolaunchctlprintsystem/com.apple.smbd
Now, test the following in the Mini’s –
Configuring the SMB Drive:
nc-vz-G5192.168.100.X445
X – represent your actual IP address (All the IP Address mentioned here are dummy in nature).
Then test the IP Address –
ping-c20192.168.100.Y
Remember, here the IP Address that is mentioned is the MacBook Pro’s IP Address i.e. Master node.
Mount the SSD from Finder
Using Command + K, you need to browse through the specific drive & navigate to the specific folder to mount it in the respective node.
smb://192.168.100.Y
Verify the SMB Drive from the terminal
mount|grepsmbfsls/Volumes
It will show the newly mounted drive from the terminal as well.
The following is directory structure of the repo for the wrapper Layer that is hosted via the designed UI –
We’ll be browsing through the right path (In our case) ->
cd/Volumes/TD_BLACK/StartUp/CollibriWrapper/scripts# Start the Master Node./start.sh
Stop The Master Node:
# To Stop the Master Node./stop.sh
Start the Worker Node:
./worker-start.sh
Stop the Worker Node:
./worker-stop.sh
So, we’ve done it. 🙂
I hope you all like this effort & let me know your feedback. I’ll be back with another topic. Until then, Happy Avenging!
Note: All the data & scenarios posted here are representative of data & scenarios available on the internet for educational purposes only. There is always room for improvement in this kind of model & the solution associated with it. This article is for educational purposes only. The techniques described should only be used for authorized security testing and research. Unauthorized access to computer systems is illegal and unethical & not encouraged.
Today, I’ll be presenting a brand new LLM that you can configure locally & based on your computation, I/O & other factors. We’re going to understand the open-source project named -> Colibri.
We’ll be presenting a series of simple & complex process, which will be partially optimized based on various scenarious.
Colibrì is an open-source inference engine for running very large Mixture-of-Experts (MoE) AI models locally. Its central idea is unusual: instead of requiring hundreds of gigabytes—or even terabytes—of model weights to fit entirely in RAM or GPU VRAM, it treats VRAM + RAM + NVMe storage as one memory hierarchy.
The router determines which experts are needed for each token, and Colibrì loads or caches those experts as required. In other words, it effectively performs just-in-time loading of model weights.
Let us understand the overall capability in a simpler form –
Capability
What Colibrì provides
Local large-model inference
Runs very large MoE models locally rather than requiring a hosted API.
Models larger than RAM/VRAM
Streams routed experts from SSD/NVMe instead of loading the full model into memory.
CPU-only operation
A GPU is not mandatory. CPU-only operation is supported.
GPU acceleration
Supports heterogeneous acceleration paths including CUDA, Metal, Vulkan, and AMD/HIP in documented configurations.
Memory multitiering
Model weights can be distributed across VRAM → RAM → NVMe according to available hardware.
Large open models
Currently documents support for nine MoE model families, including GLM, Kimi, DeepSeek, Qwen, Inkling, and OLMoE.
Local Chat interface
coli chat provides direct interactive inference.
Web UI
coli web provides a browser-based interface with performance information and model visualization.
OpenAI-compatible API
coli serve exposes endpoints such as /v1/chat/completions and /v1/models.
Anthropic-compatible API
Also provides /v1/messages, allowing some Anthropic-compatible software to communicate with Colibrì.
Tool/function calling
Available for certain supported engines, including GLM, DeepSeek V4, and Kimi K3.
Coding-tool integration
Can act as a local backend for compatible coding applications that accept an OpenAI-compatible endpoint.
Persistent KV cache
Conversation state can be persisted and reused rather than completely prefilling the model again.
Multi-machine experiments
Includes an experimental/local cluster mechanism in which other Macs can execute routed expert FFNs.
For more information, please refer to the following link.
Why large models works here?
Let us understand how this works for the larger models –
For the project’s reference GLM-5.2 model, the approximately 430 GB quantized model does not need 430 GB of RAM. The documentation describes keeping roughly 9.9 GB of dense components resident while streaming routed experts from storage as needed.
Please go throught he following link for more details.
Colibrì calls this a multitier hierarchy. Fast memory affects performance, but the design aims to avoid changing the model merely because less VRAM/RAM is available. For more information, let us understand it from the official link.
That architectural concept is probably the project’s most significant contribution.
Supported Models:
Model family
Approx. total parameters
Approx. storage/RAM characteristics documented by Colibrì
GLM-5.2 / GLM-5.3
744B
~372 GB disk; 16 GB minimum RAM
GLM-5.3-Flash
321B
~195 GB converted; ~25 GB RAM
Inkling
975B
~469 GB; ~25 GB RAM with int4 dense container
Kimi K3
2.8T
~1.6 TB disk; 32 GB+ RAM
DeepSeek V4 Flash
284B
~167 GB; 16 GB minimum
DeepSeek V4.1 Flash
552B
separate current engine implementation
Qwen3.8-Flash-Next
125B + 51B n-gram
~185.5 GB; 16 GB minimum
Qwen3.6-35B-A3B
35B / ~3B active
~20 GB; ~24 GB RAM
OLMoE
7B
~7 GB; ~8 GB RAM
The current repository specifically describes Kimi K3 at 2.8 trillion parameters, which illustrates how far the storage-streaming approach is intended to scale. One important distinction: Colibrì does not create these models. It is the inference engine that runs compatible open-weight models.
Do you need GPU to run this?
This is another major difference from frameworks primarily designed around GPU residency. The official quick-start says the baseline requirements for GLM-5.2 are approximately:
~16 GB RAM minimum, ~380 GB free storage, with fast NVMe recommended.
A GPU can significantly accelerate inference, but Colibrì’s architecture intentionally allows CPU + RAM + SSD operation. The repository currently contains backends for CUDA, Metal, Vulkan, and AMD/HIP-related configurations. Please refer the following link.
This makes Colibrì particularly interesting for heterogeneous hardware rather than only expensive data-center GPU configurations.
So, you don’t need the GPU to run any model.
Use it like a OpenAI Server:
This could be one of its most useful practical capabilities.
Run the following command:
coliserve
Creates an OpenAI-compatible HTTP API. The official documentation currently lists endpoints, including:
It supports streaming responses, token usage, temperature, top-p, maximum-token parameters, and stop sequences.
So, you can use the application either from the postman or using API-based call or from using a curl statement while invoking the following entries –
http://localhost:8000/v1
That makes Colibrì much more than a command-line model runner.
Use it like a Anthropic Server:
Interestingly, Colibrì now exposes:
/v1/messages
for software built around the Anthropic Messages API.
The documentation specifically discusses using this interface with compatible clients, including Claude Code-style integrations. For more information, please refer the following link.
Therefore, the architecture can look approximately like:
This opens some interesting possibilities for private local AI applications, coding assistants, RAG systems, research environments, and experimental agents.
Some of the models support tool/function calling. This is particularly relevent if you are thinking of agentic systems. So, Colibri potentially provides the LLM reasoing/tool-call layer of a local agent architecture.
Colibrì itself isn’t a complete agent orchestration framework like AutoGen or similar systems; it provides the model inference/API layer on which an agent framework could operate.
Colibri also offered web dashboard accessible through:
coliweb
The project describes three particularly interesting views –
Dashboard provides token-generation and performance metrics, including time breakdowns and VRAM/RAM/disk utilization.
Brain visualizes the model’s experts and shows which experts activate during generation.
Atlas visualizes measured expert specialization and routing affinities as a three-dimensional representation.
That makes Colibrì interesting not only for inference but also for studying how MoE models route information internally.
For more information, you can refer the following link.
Let us continue this topic in our next series.
Until then, Happy Avenging! 🙂
Note: All the data & scenarios posted here are representative of data & scenarios available on the internet for educational purposes only. There is always room for improvement in this kind of model & the solution associated with it. This article is for educational purposes only. The techniques described should only be used for authorized security testing and research. Unauthorized access to computer systems is illegal and unethical & not encouraged.
Today, we’ll discuss another topic in our two-part series. We will understand the importance of the MCP protocol for communicating between agents.
This will be an in-depth highly technical as well as depicting using easy-to-understand visuals.
But, before that, let us understand the demo first.
Isn’t it exciting?
MCP Protocol:
Let us first understand in easy language about the MCP protocol.
MCP (Multi-Agent Communication Protocol) is a custom message exchange system that facilitates structured and scalable communication among multiple AI agents operating within an application. These agents collaborate asynchronously or in real-time to complete complex tasks by sharing results, context, and commands through a common messaging layer.
How MCP Protocol Helps:
Feature
Benefit
Agent-Oriented Architecture
Each agent handles a focused task, improving modularity and scalability.
Event-Driven Message Passing
Agents communicate based on triggers, not polling—leading to faster and efficient responses.
Structured Communication Format
All messages follow a standard format (e.g., JSON) with metadata for sender, recipient, type, and payload.
State Preservation
Agents maintain context across messages using memory (e.g., ConversationBufferMemory) to ensure coherence.
How It Works (Step-by-Step):
📥 User uploads or streams a video.
🧑💻 MCP Protocol triggers the Transcription Agent to start converting audio into text.
🌐 Translation Agent receives this text (if a different language is needed).
🧾 Summarization Agent receives the translated or original transcript and generates a concise summary.
📚 Research Agent checks for references or terminology used in the video.
📄 Documentation Agent compiles the output into a structured report.
🔁 All communication between agents flows through MCP, ensuring consistent message delivery and coordination.
Now, let us understand the solution that we intend to implement for our solutions:
This app provides live summarization and contextual insights from videos such as webinars, interviews, or YouTube recordings using multiple cooperating AI agents. These agents may include:
Transcription Agent: Converts spoken words to text.
Translation Agent: Translates text to different languages (if needed).
Summarization Agent: Generates concise summaries.
Research Agent: Finds background or supplementary data related to the discussion.
Documentation Agent: Converts outputs into structured reports or learning materials.
We need to understand one more thing before deep diving into the code. Part of your conversation may be mixed, like part Hindi & part English. So, in that case, it will break the sentences into chunks & then convert all of them into the same language. Hence, the following rules are applied while translating the sentences –
Now, we will go through the basic frame of the system & try to understand how it fits all the principles that we discussed above for this particular solution mapped against the specific technology –
Documentation Agent built with the LangChain framework
Research Agent built with the AutoGen framework
MCP Broker for seamless communication between agents
Process Flow:
Let us understand from the given picture the flow of the process that our app is trying to implement –
Great! So, now, we’ll focus on some of the key Python scripts & go through their key features.
But, before that, we share the group of scripts that belong to specific tasks.
Message-Chaining Protocol (MCP) Implementation:
clsMCPMessage.py
clsMCPBroker.py
YouTube Transcript Extraction:
clsYouTubeVideoProcessor.py
Language Detection:
clsLanguageDetector.py
Translation Services & Agents:
clsTranslationAgent.py
clsTranslationService.py
Documentation Agent:
clsDocumentationAgent.py
Research Agent:
clsResearchAgent.py
Now, we’ll review some of the script in this post, along with the next post, as a continuation from this post.
CODE:
clsMCPMessage.py (This is one of the main or key scripts that will help enable implementation of the MCP protocols)
classclsMCPMessage(BaseModel):"""Message format for MCP protocol"""id:str=Field(default_factory=lambda:str(uuid.uuid4())) timestamp:float=Field(default_factory=time.time) sender:str receiver:str message_type:str# "request", "response", "notification" content: Dict[str, Any] reply_to: Optional[str]=None conversation_id:str metadata: Dict[str, Any]={}classclsMCPBroker:"""Message broker for MCP protocol communication between agents"""def__init__(self):self.message_queues: Dict[str, queue.Queue]={}self.subscribers: Dict[str, List[str]]={}self.conversation_history: Dict[str, List[clsMCPMessage]]={}defregister_agent(self,agent_id:str)->None:"""Register an agent with the broker"""if agent_id notinself.message_queues:self.message_queues[agent_id]= queue.Queue()self.subscribers[agent_id]=[]defsubscribe(self,subscriber_id:str,publisher_id:str)->None:"""Subscribe an agent to messages from another agent"""if publisher_id inself.subscribers:if subscriber_id notinself.subscribers[publisher_id]:self.subscribers[publisher_id].append(subscriber_id)defpublish(self,message: clsMCPMessage)->None:"""Publish a message to its intended receiver"""# Store in conversation historyif message.conversation_id notinself.conversation_history:self.conversation_history[message.conversation_id]=[]self.conversation_history[message.conversation_id].append(message)# Deliver to direct receiverif message.receiver inself.message_queues:self.message_queues[message.receiver].put(message)# Deliver to subscribers of the senderfor subscriber inself.subscribers.get(message.sender,[]):if subscriber != message.receiver:# Avoid duplicatesself.message_queues[subscriber].put(message)defget_message(self,agent_id:str,timeout: Optional[float]=None)-> Optional[clsMCPMessage]:"""Get a message for the specified agent"""try:returnself.message_queues[agent_id].get(timeout=timeout)except(queue.Empty,KeyError):returnNonedefget_conversation_history(self,conversation_id:str)-> List[clsMCPMessage]:"""Get the history of a conversation"""returnself.conversation_history.get(conversation_id,[])
Imagine a system where different virtual agents (like robots or apps) need to talk to each other. To do that, they send messages back and forth—kind of like emails or text messages. This code is responsible for:
Making sure those messages are properly written (like filling out all parts of a form).
Making sure messages are delivered to the right people.
Keeping a record of conversations so you can go back and review what was said.
This part (clsMCPMessage) is like a template or a form that every message needs to follow. Each message has:
ID: A unique number so every message is different (like a serial number).
Time Sent: When the message was created.
Sender & Receiver: Who sent the message and who is supposed to receive it.
Type of Message: Is it a request, a response, or just a notification?
Content: The actual information or question the message is about.
Reply To: If this message is answering another one, this tells which one.
Conversation ID: So we know which group of messages belongs to the same conversation.
Extra Info (Metadata): Any other small details that might help explain the message.
This (clsMCPBroker) is the system (or “post office”) that makes sure messages get to where they’re supposed to go. Here’s what it does:
1. Registering an Agent
Think of this like signing up a new user in the system.
Each agent gets their own personal mailbox (called a “message queue”) so others can send them messages.
2. Subscribing to Another Agent
If Agent A wants to receive copies of messages from Agent B, they can “subscribe” to B.
This is like signing up for B’s newsletter—whenever B sends something, A gets a copy.
3. Sending a Message
When someone sends a message:
It is saved into a conversation history (like keeping emails in your inbox).
It is delivered to the main person it was meant for.
And, if anyone subscribed to the sender, they get a copy too—unless they’re already the main receiver (to avoid sending duplicates).
4. Receiving Messages
Each agent can check their personal mailbox to see if they got any new messages.
If there are no messages, they’ll either wait for some time or move on.
5. Viewing Past Conversations
You can look up all messages that were part of a specific conversation.
This is helpful for remembering what was said earlier.
In systems where many different smart tools or services need to work together and communicate, this kind of communication system makes sure everything is:
Organized
Delivered correctly
Easy to trace back when needed
So, in this post, we’ll finish it here. We’ll cover the rest of the post in the next post.
I’ll bring some more exciting topics in the coming days from the Python verse.
Till then, Happy Avenging! 🙂
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. There is always room for improvement in this kind of model & the solution associated with it. I’ve shown the basic ways to achieve the same for educational purposes only.
As we discover in our previous post about the Sarvam AI basic capabilities & a glimpse of code review. Today, we’ll finish the rest of the part & some of the matrices comparing against other popular LLMs.
Before that, you can refer to the previous post for a recap, which is available here.
Also, we’re providing the demo here –
Now, let us jump into the rest of the code –
Code:
clsSarvamAI.py (This script will capture the audio input in Indic languages & then provide an LLM response in the form of audio in Indic languages. In this post, we’ll discuss part of the code. In the next part, we’ll be discussing the next important methods. Note that we’re only going to discuss a few important functions here.)
This method saves recorded audio data into a WAV file format.
What it Does:
Takes raw audio data and converts it into bytes.
Gets the original sample rate of the audio.
Opens a new WAV file in write mode.
Sets the parameters for the audio file (like the number of channels, sample width, and frame rate).
Writes the audio data into the file in small chunks to manage memory usage.
Logs the current time to keep track of when the audio was saved.
Returns 0 on success or 1 if there was an error.
The “createWavFile” method takes the recorded audio and saves it as a WAV file on your computer. It converts the audio into bytes and writes them into small file parts. If something goes wrong, it prints an error message.
This method breaks down a large piece of text (in Bengali) into smaller, manageable chunks.
What it Does:
Initializes an empty list to store the chunks of text.
It uses a regular expression to split the text based on punctuation marks like full stops (।), question marks (?), and exclamation points (!).
Iterates through the split sentences to form chunks that do not exceed a specified maximum length (max_length).
Adds each chunk to the list until the entire text is processed.
Returns the list of chunks or an empty string if an error occurs.
The chunkBengaliResponse method takes a long Bengali text and splits it into smaller, easier-to-handle parts. It uses punctuation marks to determine where to split. If there’s a problem while splitting, it prints an error message.
This method plays audio data stored in a WAV file format.
What it Does:
Reads the audio data from a WAV file object.
Extracts parameters like the number of channels, sample width, and frame rate.
Converts the audio data into a format that the sound device can process.
If the audio is stereo (two channels), it reshapes the data for playback.
Plays the audio through the speakers.
Returns 0 on success or 1 if there was an error.
The playWav method takes audio data from a WAV file and plays it through your computer’s speakers. It reads the data and converts it into a format your speakers can understand. If there’s an issue playing the audio, it prints an error message.
This method continuously plays audio from a queue until there is no more audio to play.
What it Does:
It enters an infinite loop to keep checking for audio data in the queue.
Retrieves audio data from the queue and plays it using the “playWav”-method.
Logs the current time each time an audio response is played.
It breaks the loop if it encounters a None value, indicating no more audio to play.
Returns 0 on success or 1 if there was an error.
The audioPlayerWorker method keeps checking a queue for new audio to play. It plays each piece of audio as it comes in and stops when there’s no more audio. If there’s an error during playback, it prints an error message.
This asynchronous method processes a chunk of text to generate audio using an external API.
What it Does:
Cleans up the text chunk by removing unwanted characters.
Prepares a payload with the cleaned text and other parameters required for text-to-speech conversion.
Sends a POST request to an external API to generate audio from the text.
Decodes the audio data received from the API (in base64 format) into raw audio bytes.
Returns the audio bytes or an empty byte string if there is an error.
The processChunk method takes a text, sends it to an external service to be converted into speech, and returns the audio data. If something goes wrong, it prints an error message.
This asynchronous method handles the complete audio processing workflow, including speech recognition, translation, and audio playback.
What it Does:
Initializes various configurations and headers required for processing.
Sends the recorded audio to an API to get the transcript and detected language.
Translates the transcript into another language using another API.
Splits the translated text into smaller chunks using the chunkBengaliResponse method.
Starts an audio playback thread to play each processed audio chunk.
Sends each text chunk to the processChunk method to convert to speech and adds the audio data to the queue for playback.
Waits for all audio chunks to be processed and played before finishing.
Logs the current time when the process is complete.
Returns 0 on success or 1 if there was an error.
The “processAudio”-method takes recorded audio, recognizes what was said, translates it into another language, splits the translated text into parts, converts each part into speech, and plays it back. It uses different services to do this; if there’s a problem at any step, it prints an error message.
And, here is the performance stats (Captured from Sarvam AI website) –
So, finally, we’ve done it. You can view the complete code in this GitHub link.
I’ll bring some more exciting topics in the coming days from the Python verse.
Till then, Happy Avenging! 🙂
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. There is always room for improvement in this kind of model & the solution associated with it. I’ve shown the basic ways to achieve the same for educational purposes only.
In the rapidly evolving landscape of artificial intelligence, Sarvam AI has emerged as a pioneering force in developing language technologies for Indian languages. This article series aims to provide an in-depth look at Sarvam AI’s Indic APIs, exploring their features, performance, and potential impact on the Indian tech ecosystem.
This LLM aims to bridge the language divide in India’s digital landscape by providing powerful, accessible AI tools for Indic languages.
The importance of Indic language processing:
India has 22 official languages and hundreds of dialects, presenting a unique challenge for technology adoption and digital inclusion. Even though all the government work happens in both the official language along with English language.
Developers can fine-tune the models for specific domains or use cases, improving accuracy for specialized applications.
Supported languages and use cases:
As of 2024, Sarvam AI’s Indic APIs support the following languages:
Hindi
Bengali
Tamil
Telugu
Marathi
Gujarati
Kannada
Malayalam
Punjabi
Odia
Before delving into the details, I strongly recommend taking a look at the demo.
Isn’t this exciting? Let us understand the flow of events in the following diagram –
The application interacts with Sarvam AI’s API. After interpreting the initial audio inputs from the computer, it uses Sarvam AI’s API to get the answer based on the selected Indic language, Bengali.
clsSarvamAI.py (This script will capture the audio input in Indic languages & then provide an LLM response in the form of audio in Indic languages. In this post, we’ll discuss part of the code. In the next part, we’ll be discussing the next important methods. Note that we’re only going to discuss a few important functions here.)
definitializeMicrophone(self):try:forindex,nameinenumerate(sr.Microphone.list_microphone_names()):print(f"Microphone with name \"{name}\" found (device_index={index})")returnsr.Microphone()exceptExceptionas e:x=str(e)print('Error: <<Initiating Microphone>>: ',x)return''defrealTimeTranslation(self):try:WavFile=self.WavFilerecognizer=sr.Recognizer()try:microphone=self.initializeMicrophone()exceptExceptionas e:print(f"Error initializing microphone: {e}")returnwithmicrophoneas source:print("Adjusting for ambient noise. Please wait...")recognizer.adjust_for_ambient_noise(source,duration=5)print("Microphone initialized. Start speaking...")try:while True:try:print("Listening...")audio=recognizer.listen(source,timeout=5,phrase_time_limit=5)print("Audio captured. Recognizing...") #var=datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S") #print('Before Audio Time: ',str(var))self.createWavFile(audio,WavFile)try:text=recognizer.recognize_google(audio,language="bn-BD") # Bengalilanguagecodesentences=text.split('।') # Bengalifullstopprint('Sentences: ')print(sentences)print('*'*120)ifnot text:print("No speech detected. Please try again.")continueifstr(text).lower() =='টাটা':raiseBreakOuterLoop("Based on User Choice!")asyncio.run(self.processAudio(audio))exceptsr.UnknownValueError:print("Google Speech Recognition could not understand audio")exceptsr.RequestErroras e:print(f"Could not request results from Google Speech Recognition service; {e}")exceptsr.WaitTimeoutError:print("No speech detected within the timeout period. Listening again...")except BreakOuterLoop:raiseexceptExceptionas e:print(f"An unexpected error occurred: {e}")time.sleep(1) # ShortpausebeforenextiterationexceptBreakOuterLoopas e:print(f"Exited : {e}") # Removingthetemporaryaudiofilethatwasgeneratedatthebeginingos.remove(WavFile)return0exceptExceptionas e:x=str(e)print('Error: <<Real-time Translation>>: ',x)return1
initializeMicrophone:
Purpose:
This method is responsible for setting up and initializing the microphone for audio input.
What it Does:
It attempts to list all available microphones connected to the system.
It prints the microphone’s name and corresponding device index (a unique identifier) for each microphone.
If successful, it returns a microphone object (sr.Microphone()), which can be used later to capture audio.
If this process encounters an error (e.g., no microphones being found or an internal error), it catches the exception, prints an error message, and returns an empty string (“).
The “initializeMicrophone” Method finds all microphones connected to the computer and prints their names. If it finds a microphone, it prepares to use it for recording. If something goes wrong, it tells you what went wrong and stops the process.
realTimeTranslation:
Purpose: This Method uses the microphone to handle real-time speech translation from a user. It captures spoken audio, converts it into text, and processes it further.
What it Does:
Initializes a recognizer object (sr.Recognizer()) for speech recognition.
Call initializeMicrophone to set up the microphone. If initialization fails, an error message is printed, and the process is stopped.
Once the microphone is set up successfully, it adjusts for ambient noise to enhance accuracy.
Enters a loop to continuously listen for audio input from the user:
It waits for the user to speak and captures the audio.
Converts the captured audio to text using Google’s Speech Recognition service, specifying Bengali as the language.
If text is successfully captured and recognized:
Splits the text into sentences using the Bengali full-stop character.
Prints the sentences.
It checks if the text is a specific word (“টাটা”), and if so, it raises an exception to stop the loop (indicating that the user wants to exit).
Otherwise, it processes the audio asynchronously with processAudio.
If no speech is detected or an error occurs, it prints the relevant message and continues listening.
If the user decides to exit or if an error occurs, it breaks out of the loop, deletes any temporary audio files created, and returns a status code (0 for success, 1 for failure).
The “realTimeTranslation” method continuously listens to the microphone for the user to speak. It captures what is said and tries to understand it using Google’s service, specifically for the Bengali language. It then splits what was said into sentences and prints them out. If the user says “টাটা” (which means “goodbye” in Bengali), it stops listening and exits. If it cannot understand the user or if there is a problem, it will let the user know and try again. It will print an error and stop the process if something goes wrong.
Let’s wait for the next part & enjoy this part.
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. There is always room for improvement in this kind of model & the solution associated with it. I’ve shown the basic ways to achieve the same for educational purposes only.
This new solution will evaluate the power of Stable Defussion, which is created solutions as we progress & refine our prompt from scratch by using Stable Defussion & Python. This post opens new opportunities for IT companies & business start-ups looking to deliver solutions & have better performance compared to the paid version of Stable Defussion AI’s API performance. This project is for the advanced Python, Stable Defussion for data Science Newbies & AI evangelists.
In a series of posts, I’ll explain and focus on the Stable Defussion API and custom solution using the Python-based SDK of Stable Defussion.
But, before that, let us view the video that it generates from the prompt by using the third-party API:
Prompt to Video
And, let us understand the prompt that we supplied to create the above video –
Lighthouse on a cliff overlooking the ocean, dynamic ocean waves crashing against rocks, dramatic clouds moving across sky, photorealistic water movement, mist and ocean spray, wind-driven waves, atmospheric sky motion, natural fluid dynamics, realistic, detailed, 8k. Do not change the size & shape of the lighthouse & the field on top of which the Lighthouse built.
Isn’t it exciting?
However, I want to stress this point: the video generated by the Stable Defusion (Stability AI) API was able to partially apply the animation effect. Even though the animation applies to the cloud, It doesn’t apply the animation to the wave. But, I must admit, the quality of the video is quite good.
Let us understand the code and how we run the solution, and then we can try to understand its performance along with the other solutions later in the subsequent series.
As you know, we’re exploring the code base of the third-party API, which will actually execute a series of API calls that create a video out of the prompt.
CODE:
Let us understand some of the important snippet –
classclsStabilityAIAPI:def__init__(self, STABLE_DIFF_API_KEY, OUT_DIR_PATH, FILE_NM, VID_FILE_NM):self.STABLE_DIFF_API_KEY = STABLE_DIFF_API_KEYself.OUT_DIR_PATH = OUT_DIR_PATHself.FILE_NM = FILE_NMself.VID_FILE_NM = VID_FILE_NMdefdelFile(self, fileName):try: # Deletingtheintermediateimageos.remove(fileName)return 0 exceptExceptionase:x = str(e)print('Error: ', x)return 1defgenerateText2Image(self, inputDescription):try:STABLE_DIFF_API_KEY = self.STABLE_DIFF_API_KEYfullFileName = self.OUT_DIR_PATH + self.FILE_NMifSTABLE_DIFF_API_KEYisNone:raiseException("MissingStabilityAPIkey.")response = requests.post(f"{api_host}/v1/generation/{engine_id}/text-to-image",headers={"Content-Type":"application/json","Accept":"application/json","Authorization":f"Bearer {STABLE_DIFF_API_KEY}"},json={"text_prompts": [{"text":inputDescription}],"cfg_scale":7,"height":1024,"width":576,"samples":1,"steps":30,},)ifresponse.status_code!=200:raiseException("Non-200 response: "+str(response.text))data=response.json()fori,imageinenumerate(data["artifacts"]):withopen(fullFileName,"wb") as f:f.write(base64.b64decode(image["base64"])) returnfullFileNameexceptExceptionas e:x=str(e)print('Error: ',x)return'N/A'defimage2VideoPassOne(self,imgNameWithPath):try:STABLE_DIFF_API_KEY=self.STABLE_DIFF_API_KEYresponse=requests.post(f"https://api.stability.ai/v2beta/image-to-video",headers={"authorization":f"Bearer {STABLE_DIFF_API_KEY}"},files={"image":open(imgNameWithPath,"rb")},data={"seed":0,"cfg_scale":1.8,"motion_bucket_id":127}, )print('First Pass Response:')print(str(response.text))genID=response.json().get('id')returngenIDexceptExceptionas e:x=str(e)print('Error: ',x)return'N/A'defimage2VideoPassTwo(self,genId):try:generation_id=genIdSTABLE_DIFF_API_KEY=self.STABLE_DIFF_API_KEYfullVideoFileName=self.OUT_DIR_PATH+self.VID_FILE_NMresponse=requests.request("GET",f"https://api.stability.ai/v2beta/image-to-video/result/{generation_id}",headers={'accept':"video/*", # Use 'application/json' to receive base64 encoded JSON'authorization':f"Bearer {STABLE_DIFF_API_KEY}"},) print('Retrieve Status Code: ',str(response.status_code))ifresponse.status_code==202:print("Generation in-progress, try again in 10 seconds.")return5elifresponse.status_code==200:print("Generation complete!")withopen(fullVideoFileName,'wb') as file:file.write(response.content)print("Successfully Retrieved the video file!")return0else:raiseException(str(response.json()))exceptExceptionas e:x=str(e)print('Error: ',x)return1
Now, let us understand the code –
1. CLASS INSTANTIATION
This function is called when an object of the class is created. It initializes four properties:
STABLE_DIFF_API_KEY: the API key for Stability AI services.
OUT_DIR_PATH: the folder path to save files.
FILE_NM: the name of the generated image file.
VID_FILE_NM: the name of the generated video file.
2. delFile(fileName)
This function deletes a file specified by fileName.
If successful, it returns 0.
If an error occurs, it logs the error and returns 1.
3. generateText2Image(inputDescription)
This function generates an image based on a text description:
Sends a request to the Stability AI text-to-image endpoint using the API key.
Saves the resulting image to a file.
Returns the file’s path on success or 'N/A' if an error occurs.
4. image2VideoPassOne(imgNameWithPath)
This function uploads an image to create a video in its first phase:
Sends the image to Stability AI’s image-to-video endpoint.
Logs the response and extracts the id (generation ID) for the next phase.
Returns the id if successful or 'N/A' on failure.
5. image2VideoPassTwo(genId)
This function retrieves the video created in the second phase using the genId:
Checks the video generation status from the Stability AI endpoint.
If complete, saves the video file and returns 0.
If still processing, returns 5.
Logs and returns 1 for any errors.
As you can see, the code is pretty simple to understand & we’ve taken all the necessary actions in case of any unforeseen network issues or even if the video is not ready after our job submission in the following lines of the main calling script (generateText2VideoAPI.py) –
Please let me know your feedback after reviewing all the posts! 🙂
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. There is always room for improvement in this kind of model & the solution associated with it. I’ve shown the basic ways to achieve the same for educational purposes only.
Today, we’ll share the second installment of the RAG implementation. If you are new here, please visit the previous post for full context.
In this post, we’ll be discussing the Haystack framework more. Again, before discussing the main context, I want to present the demo here.
Demo
FLOW OF EVENTS:
Let us look at the flow diagram as it captures the sequence of events that unfold as part of the process, where today, we’ll pay our primary attention.
As you can see today, we’ll discuss the red dotted line, which contextualizes the source data into the Vector DBs.
Let us understand the flow of events here –
The main Python application will consume the nested JSON by invoking the museum API in multiple threads.
The application will clean the nested data & extract the relevant attributes after flattening the JSON.
It will create the unstructured text-based context, which is later fed to the Vector DB framework.
We’re using the Metropolitan Museum API to feed the data to our Vector DB. For more information, please visit the following link. And this is free to use & moreover, we’re using it for education scenarios.
CODE:
We’ll discuss the tokenization part highlighted in a red dotted line from the above picture.
Python:
We’ll discuss the scripts in the diagram as part of the flow mentioned above.
clsExtractJSON.py (This is the main class that will extract the content from the museum API using parallel calls.)
The above code translates into the following steps –
The above method first calls the generateFirstDayOfLastTenYears() plan to populate records for every department after getting all the unique departments by calling another API.
Then, it will call the getDataThread() methods to fetch all the relevant APIs simultaneously to reduce the overall wait time & create individual smaller files.
Finally, the application will invoke the mergeCsvFilesInDirectory() method to merge all the chunk files into one extensive historical data.
The above method will merge all the small files into a single, more extensive historical data that contains over ten years of data (the first day of ten years of data, to be precise).
For the complete code, please visit the GitHub.
1_ReadMuseumJSON.py (This is the main class that will invoke the class, which will extract the content from the museum API using parallel calls.)
The above script calls the main class after instantiating the class.
clsCreateList.py (This is the main class that will extract the relevant attributes from the historical files & then create the right input text to create the documents for contextualize into the Vector DB framework.)
The above code will read the data from the extensive historical file created from the earlier steps & then it will clean the file by removing all the duplicate records (if any) & finally, it will create three unique URLs that constitute artist, object & wiki.
Also, this application will remove the hyperlink with a specific hash value, which will feed into the vector DB. Vector DB could be better with the URLs. Hence, we will store the URLs in a separate file by storing the associate hash value & later, we’ll fetch it in a lookup from the open AI response.
Then, this application will generate prompts dynamically & finally create the documents for later steps of vector DB consumption by invoking the addDocument() methods.
For more details, please visit the GitHub link.
1_1_testCreateRec.py (This is the main class that will call the above class.)
In the above script, the following essential steps took place –
First, the application calls the clsCreateList class to store all the documents inside a dictionary.
Then it stores the data inside the vector DB & creates & stores the model, which will be later reused (If you remember, we’ve used this as a model in our previous post).
Finally, test with some sample use cases by providing the proper context to OpenAI & confirm the response.
Here is a short clip of how the RAG models contextualize with the source data.
RAG-ModelContextualization
So, finally, we’ve done it.
I know that this post is relatively bigger than my earlier post. But, I think, you can get all the details once you go through it.
You will get the complete codebase in the following GitHub link.
I’ll bring some more exciting topics in the coming days from the Python verse. Please share & subscribe to my post & let me know your feedback.
Till then, Happy Avenging! 🙂
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. Some of the images (except my photo) we’ve used are available over the net. We don’t claim ownership of these images. There is always room for improvement & especially in the prediction quality.
Today, I will share a new post in a part series about creating end-end LLMs that feed source data with RAG implementation. I’ll also use OpenAI python-based SDK and Haystack embeddings in this case.
In this post, I’ve directly subscribed to OpenAI & I’m not using OpenAI from Azure. However, I’ll explore that in the future as well.
Before I explain the process to invoke this new library, why not view the demo first & then discuss it?
Demo
FLOW OF EVENTS:
Let us look at the flow diagram as it captures the sequence of events that unfold as part of the process.
As you can see, to enable this large & complex solution, we must first establish the capabilities to build applications powered by LLMs, Transformer models, vector search, and more. You can use state-of-the-art NLP models to perform question-answering, answer generation, semantic document search, or build tools capable of complex decision-making and query resolution. Hence, steps no. 1 & 2 showcased the data embedding & creating that informed repository. We’ll be discussing that in our second part.
Once you have the informed repository, the system can interact with the end-users. As part of the query (shown in step 3), the prompt & the question are shared with the process engine, which then turned to reduce the volume & get relevant context from our informed repository & get the tuned context as part of the response (Shown in steps 4, 5 & 6).
Then, this tuned context is shared with the OpenAI for better response & summary & concluding remarks that are very user-friendly & easier to understand for end-users (Shown in steps 8 & 9).
IMPORTANT PACKAGES:
The following are the important packages that are essential to this project –
Let us understand some of the important sections of the above script –
Function – login():
The login function retrieves a ‘username’ and ‘password’ from a JSON request and prints them. It checks if the provided credentials are missing from users or password lists, returning a failure JSON response if so. It creates and returns an access token in a JSON response if valid.
Function – get_chat():
The get_chat function retrieves the running session count and user input from a JSON request. Based on the session count, it extracts catalog data or processes the user’s message from the RAG framework that finally receives the refined response from the OpenAI, extracting hash values, image URLs, and wiki URLs. If an error arises, the function captures and returns the error as a JSON message.
Function – updateCounter():
The updateCounter function checks if a given CSV file exists and retrieves its counter value. It then increments the counter and writes it back to the CSV. If any errors occur, an error message is printed, and the function returns a value of 1.
Function – extractRemoveUrls():
The extractRemoveUrls function attempts to filter a data frame, resDf, based on a provided hash value to extract image and wiki URLs. If the data frame contains matching entries, it retrieves the corresponding URLs. Any errors encountered are printed, but the function always returns the image and wiki URLs, even if they are empty.
clsContentScrapper.py (This is the main class that brings the default options for the users if they agree with the initial prompt by the bot.)
Let us understand the the core part that require from this class.
Function – extractCatalog():
The extractCatalog function uses specific headers to make a GET request to a constructed URL. The URL is derived by appending ‘/departments’ to a base_url, and a header token is used in the request headers. If successful, it returns the text of the response; if there’s an exception, it prints the error and returns the error message.
clsRAGOpenAI.py (This is the main class that brings the RAG-enabled context that is fed to OpenAI for fine-tuned response with less cost.)
############################################################# Written By:SATYAKIDE ######## Written On:27-Jun-2023 ######## ModifiedOn28-Jun-2023 ######## ######## Objective:Thisisthemaincalling ######## pythonscriptthatwillinvokethe ######## shortcutapplicationcreatedinsideMAC ######## enviornmentincludingMacBook,IPadorIPhone. ######## #############################################################fromhaystack.document_stores.faissimportFAISSDocumentStorefromhaystack.nodesimportDensePassageRetrieverimportopenaifromclsConfigClientimportclsConfigClientascfimportclsLaslog# DisblingWarningdefwarn(*args,**kwargs):passimportwarningswarnings.warn = warnimportosimportre################################################## GlobalSection ##################################################Ind = cf.conf['DEBUG_IND']queryModel = cf.conf['QUERY_MODEL']passageModel = cf.conf['PASSAGE_MODEL']#InitiatingLoggingInstancesclog = log.clsL()os.environ["TOKENIZERS_PARALLELISM"] = "false"vectorDBFileName = cf.conf['VECTORDB_FILE_NM']indexFile = "vectorDB/" + str(vectorDBFileName) + '.faiss'indexConfig = "vectorDB/" + str(vectorDBFileName) + ".json"print('File: ',str(indexFile))print('Config: ',str(indexConfig))# Also,provide`config_path`parameterifyousetitwhencallingthe`save()`method:new_document_store = FAISSDocumentStore.load(index_path=indexFile,config_path=indexConfig)# InitializeRetrieverretriever = DensePassageRetriever(document_store=new_document_store,query_embedding_model=queryModel,passage_embedding_model=passageModel,use_gpu=False)################################################## EndofGlobalSection ##################################################classclsRAGOpenAI:def__init__(self):self.basePath = cf.conf['DATA_PATH']self.fileName = cf.conf['FILE_NAME']self.Ind = cf.conf['DEBUG_IND']self.subdir = str(cf.conf['OUT_DIR'])self.base_url = cf.conf['BASE_URL']self.outputPath = cf.conf['OUTPUT_PATH']self.vectorDBPath = cf.conf['VECTORDB_PATH']self.openAIKey = cf.conf['OPEN_AI_KEY']self.temp = cf.conf['TEMP_VAL']self.modelName = cf.conf['MODEL_NAME']self.maxToken = cf.conf['MAX_TOKEN']defextractHash(self,text):try: # Regularexpressionpatterntomatch'Ref: {'followedbyanumberandthen'}'pattern = r"Ref: \{'(\d+)'\}"match = re.search(pattern,text)ifmatch:returnmatch.group(1)else:returnNoneexceptExceptionase:x = str(e)print('Error: ',x)returnNonedefremoveSentencesWithNaN(self,text):try: # Splittextintosentencesusingregularexpressionsentences = re.split('(?<!\w\.\w.)(?<![A-Z][a-z]\.)(?<=\.|\?)\s',text) # Filteroutsentencescontaining'nan'filteredSentences = [sentenceforsentenceinsentencesif'nan'notinsentence] # Rejointhesentencesreturn''.join(filteredSentences)exceptExceptionase:x = str(e)print('Error: ',x)return''defretrieveDocumentsReader(self,question,top_k=9):returnretriever.retrieve(question,top_k=top_k)defgenerateAnswerWithGPT3(self,retrieved_docs,question):try:openai.api_key = self.openAIKeytemp = self.tempmodelName = self.modelNamemaxToken = self.maxTokendocumentsText = "".join([doc.contentfordocinretrieved_docs])filteredDocs = self.removeSentencesWithNaN(documentsText)hashValue = self.extractHash(filteredDocs)print('RAG Docs:: ')print(filteredDocs) #prompt = f"Given the following documents: {documentsText}, answer the question accurately based on the above data with the supplied http urls: {question}" # Setupachat-stylepromptwithyourdatamessages = [{"role": "system", "content": "Youareahelpfulassistant,answerthequestionaccuratelybasedontheabovedatawiththesuppliedhttpurls. Onlyrelevantcontentneedstopublish. Pleasedonotprovidethefactsorthetextsthatresultscrossingthemax_tokenlimits."},{"role": "user", "content": filteredDocs} ] # Chatstyleinvokingthelatestmodelresponse = openai.ChatCompletion.create(model=modelName,messages=messages,temperature = temp,max_tokens=maxToken )returnhashValue,response.choices[0].message['content'].strip().replace('\n','\\n')exceptExceptionase:x = str(e)print('failed to get from OpenAI: ',x)return'Not Available!'defragAnswerWithHaystackAndGPT3(self,question):retrievedDocs = self.retrieveDocumentsReader(question)returnself.generateAnswerWithGPT3(retrievedDocs,question)defgetData(self,strVal):try:print('*'*120)print('Index Your Data for Retrieval:')print('*'*120)print('Response from New Docs: ')print()hashValue,answer = self.ragAnswerWithHaystackAndGPT3(strVal)print('GPT3 Answer::')print(answer)print('Hash Value:')print(str(hashValue))print('*'*240)print('End Of Use RAG to Generate Answers:')print('*'*240)returnhashValue,answerexceptExceptionase:x = str(e)print('Error: ',x)answer = xhashValue = 1returnhashValue,answer
Let us understand some of the important block –
Function – ragAnswerWithHaystackAndGPT3():
The ragAnswerWithHaystackAndGPT3 function retrieves relevant documents for a given question using the retrieveDocumentsReader method. It then generates an answer for the query using GPT-3 with the retrieved documents via the generateAnswerWithGPT3 method. The final response is returned.
Function – generateAnswerWithGPT3():
The generateAnswerWithGPT3 function, given a list of retrieved documents and a question, communicates with OpenAI’s GPT-3 to generate an answer. It first processes the documents, filtering and extracting a hash value. Using a chat-style format, it prompts GPT-3 with the processed documents and captures its response. If an error occurs, an error message is printed, and “Not Available!” is returned.
Function – retrieveDocumentsReader():
The retrieveDocumentsReader function takes in a question and an optional parameter, top_k (defaulted to 9). It is called the retriever.retrieve method with the given parameters. The result of the retrieval will generate at max nine responses from the RAG engine, which will be fed to OpenAI.
React:
App.js (This is the main react script, that will create the interface & parse the data apart from the authentication)
// App.jsimportReact,{useState}from'react';importaxiosfrom'axios';import'./App.css';constApp=()=>{const[isLoggedIn,setIsLoggedIn]=useState(false);const[username,setUsername]=useState('');const[password,setPassword]=useState('');const[message,setMessage]=useState('');const[chatLog,setChatLog]=useState([{sender:'MuBot',message:'Welcome to MuBot! Please explore the world of History from our brilliant collections! Do you want to proceed to see the catalog?'}]);consthandleLogin=async(e)=>{e.preventDefault();try{constresponse=awaitaxios.post('http://localhost:5000/login',{username,password});if (response.status===200) {setIsLoggedIn(true);}}catch (error) {console.error('Login error:',error);}};constsendMessage=async(username)=>{if (message.trim() ==='') return;// Create a new chat entryconstnewChatEntry={sender:'user',message:message.trim(),};// Clear the input fieldsetMessage('');try{// Make API request to Python-based APIconstresponse=awaitaxios.post('http://localhost:5000/chat',{message:newChatEntry.message});// Replace with your API endpoint URLconstresponseData=response.data;// Print the response to the console for debuggingconsole.log('API Response:',responseData);// Parse the nested JSON from the 'message' attributeconstjsonData=JSON.parse(responseData.message);// Check if the data contains 'departments'if (jsonData.departments) {// Extract the 'departments' attribute from the parsed dataconstdepartments=jsonData.departments;// Extract the department names and create a single string with line breaksconstbotResponseText=departments.reduce((acc,department)=>{returnacc+department.departmentId+''+department.displayName+'\n';},'');// Update the chat log with the bot's responsesetChatLog((prevChatLog)=> [...prevChatLog,{sender:'user',message:message},{sender:'bot',message:botResponseText},]);}elseif (jsonData.records){// Data structure 2: Artwork informationconstrecords=jsonData.records;// Prepare chat entriesconstchatEntries= [];// Iterate through records and extract text, image, and wiki informationrecords.forEach((record)=>{consttextInfo=Object.entries(record).map(([key,value])=>{if (key!=='Image'&&key!=='Wiki') {return`${key}: ${value}`;}returnnull;}).filter((info)=>info!==null).join('\n');constimageLink=record.Image;//const wikiLinks = JSON.parse(record.Wiki.replace(/'/g, '"'));//const wikiLinks = record.Wiki;constwikiLinks=record.Wiki.split(',').map(link=>link.trim());console.log('Wiki:',wikiLinks);// Check if there is a valid image linkconsthasValidImage=imageLink&&imageLink!=='[]';constimageElement=hasValidImage? (<imgsrc={imageLink}alt="Artwork"style={{maxWidth:'100%'}}/> ) :null;// Create JSX elements for rendering the wiki links (if available)constwikiElements=wikiLinks.map((link,index)=> (<divkey={index}><ahref={link}target="_blank"rel="noopener noreferrer"> Wiki Link {index+1}</a></div> ));if (textInfo) {chatEntries.push({sender:'bot',message:textInfo});}if (imageElement) {chatEntries.push({sender:'bot',message:imageElement});}if (wikiElements.length >0) {chatEntries.push({sender:'bot',message:wikiElements});}});// Update the chat log with the bot's responsesetChatLog((prevChatLog)=> [...prevChatLog,{sender:'user',message},...chatEntries, ]);}}catch (error) {console.error('Error sending message:',error);}};if (!isLoggedIn) {return (<divclassName="login-container"><h2>Welcome to the MuBot</h2><formonSubmit={handleLogin}className="login-form"><inputtype="text"placeholder="Enter your name"value={username}onChange={(e)=>setUsername(e.target.value)}required/><inputtype="password"placeholder="Enter your password"value={password}onChange={(e)=>setPassword(e.target.value)}required/><buttontype="submit">Login</button></form></div> );}return (<divclassName="chat-container"><divclassName="chat-header"><h2>Hello, {username}</h2><h3>Chat with MuBot</h3></div><divclassName="chat-log">{chatLog.map((chatEntry,index)=> (<divkey={index}className={`chat-entry ${chatEntry.sender==='user'?'user':'bot'}`}><spanclassName="user-name">{chatEntry.sender==='user'?username:'MuBot'}</span><pclassName="chat-message">{chatEntry.message}</p></div> ))}</div><divclassName="chat-input"><inputtype="text"placeholder="Type your message..."value={message}onChange={(e)=>setMessage(e.target.value)}onKeyPress={(e)=>{if (e.key==='Enter') {sendMessage();}}}/><buttononClick={sendMessage}>Send</button></div></div> );};exportdefaultApp;
Please find some of the important logic –
Function – handleLogin():
The handleLogin asynchronous function responds to an event by preventing its default action. It attempts to post a login request with a username and password to a local server endpoint. If the response is successful with a status of 200, it updates a state variable to indicate a successful login; otherwise, it logs any encountered errors.
Function – sendMessage():
The sendMessage asynchronous function is designed to handle the user’s chat interaction:
If the message is empty (after trimming spaces), the function exits without further action.
A chat entry object is created with the sender set as ‘user’ and the trimmed message.
The input field’s message is cleared, and an API request is made to a local server endpoint with the chat message.
If the API responds with a ‘departments’ attribute in its JSON, a bot response is crafted by iterating over department details.
If the API responds with ‘records’ indicating artwork information, the bot crafts responses for each record, extracting text, images, and wiki links, and generating JSX elements for rendering them.
After processing the API response, the chat log state is updated with the user’s original message and the bot’s responses.
Errors, if encountered, are logged to the console.
This function enables interactive chat with bot responses that vary based on the nature of the data received from the API.
DIRECTORY STRUCTURES:
Let us explore the directory structure starting from the parent to some of the important child folder should look like this –
So, finally, we’ve done it.
I know that this post is relatively bigger than my earlier post. But, I think, you can get all the details once you go through it.
You will get the complete codebase in the following GitHub link.
I’ll bring some more exciting topics in the coming days from the Python verse. Please share & subscribe to my post & let me know your feedback.
Till then, Happy Avenging! 🙂
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. Some of the images (except my photo) we’ve used are available over the net. We don’t claim ownership of these images. There is always room for improvement & especially in the prediction quality.
This week we’re going to extend one of our earlier posts & trying to read an entire text from streaming using computer vision. If you want to view the previous post, please click the following link.
But, before we proceed, why don’t we view the demo first?
Demo
Architecture:
Let us understand the architecture flow –
Architecture flow
The above diagram shows that the application, which uses the Open-CV, analyzes individual frames from the source & extracts the complete text within the video & displays it on top of the target screen besides prints the same in the console.
Let us now understand the code. For this use case, we will only discuss three python scripts. However, we need more than these three. However, we have already discussed them in some of the early posts. Hence, we will skip them here.
clsReadingTextFromStream.py (This is the main class of python script that will extract the text from the WebCAM streaming in real-time.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Please find the key snippet from the above script –
# Two output layer names for the text detector model
lNames = cf.conf['LAYER_DET']
# Tesseract OCR text param values
strVal = "-l " + str(cf.conf['LANG']) + " --oem " + str(cf.conf['OEM_VAL']) + " --psm " + str(cf.conf['PSM_VAL']) + ""
config = (strVal)
The first line contains the two output layers’ names for the text detector model. Among them, the first one indicates the outcome possibilities & the second one use to derive the bounding box coordinates of the predicted text.
The second line contains various options for the tesseract APIs. You need to understand the opportunities in detail to make them work. These are the essential options for our use case –
Language – The intended language, for example, English, Spanish, Hindi, Bengali, etc.
OEM flag – In this case, the application will use 4 to indicate LSTM neural net model for OCR.
OEM Value – In this case, the selected value is 7, indicating that the application treats the ROI as a single line of text.
For more details, please refer to the config file.
print("[INFO] Loading Text Detector...")
net = cv2.dnn.readNet(modelPath)
The above lines bring the already created model & load it to memory for evaluation.
# Setting new width and height and then determine the ratio in change
# for both the width and height
(newW, newH) = (wt, ht)
rW = origW / float(newW)
rH = origH / float(newH)
# Resize the frame and grab the new frame dimensions
frame = cv2.resize(frame, (newW, newH))
(H, W) = frame.shape[:2]
# Construct a blob from the frame and then perform a forward pass of
# the model to obtain the two output layer sets
blob = cv2.dnn.blobFromImage(frame, 1.0, (W, H), sParam, swapRB=True, crop=False)
net.setInput(blob)
(confScore, imgGeo) = net.forward(lNames)
# Decode the predictions, then apply non-maxima suppression to
# suppress weak, overlapping bounding boxes
(rects, confidences) = self.predictText(confScore, imgGeo)
boxes = non_max_suppression(np.array(rects), probs=confidences)
The above lines are more of preparing individual frames to get the bounding box by resizing the height & width followed by a forward pass of the model to obtain two output layer sets. And then apply the non-maxima suppression to remove the weak, overlapping bounding box by interpreting the prediction. In short, this will identify the potential text region & put the bounding box surrounding it.
# Initialize the list of results
res = []
# Getting BoundingBox boundaries
res = self.findBoundBox(boxes, res, rW, rH, orig, origW, origH, pad)
The above function will create the bounding box surrounding the predicted text regions. Also, we will capture the expected text inside the result variable.
for (spX, spY, epX, epY) in boxes:
# Scale the bounding box coordinates based on the respective
# ratios
spX = int(spX * rW)
spY = int(spY * rH)
epX = int(epX * rW)
epY = int(epY * rH)
# To obtain a better OCR of the text we can potentially
# apply a bit of padding surrounding the bounding box.
# And, computing the deltas in both the x and y directions
dX = int((epX - spX) * pad)
dY = int((epY - spY) * pad)
# Apply padding to each side of the bounding box, respectively
spX = max(0, spX - dX)
spY = max(0, spY - dY)
epX = min(origW, epX + (dX * 2))
epY = min(origH, epY + (dY * 2))
# Extract the actual padded ROI
roi = orig[spY:epY, spX:epX]
Now, the application will scale the bounding boxes based on the previously computed ratio for actual text recognition. In this process, the application also padded the bounding boxes & then extracted the padded region of interest.
# Choose the proper OCR Config
text = pytesseract.image_to_string(roi, config=config)
# Add the bounding box coordinates and OCR'd text to the list
# of results
res.append(((spX, spY, epX, epY), text))
Using OCR options, the application extracts the text within the video frame & adds that to the res list.
# Sort the results bounding box coordinates from top to bottom
res = sorted(res, key=lambda r:r[0][1])
It then sends a sorted output to the primary calling functions.
for ((spX, spY, epX, epY), text) in res:
# Display the text OCR by using Tesseract APIs
print("Reading Text::")
print("=" *60)
print(text)
print("=" *60)
# Removing the non-ASCII text so it can draw the text on the frame
# using OpenCV, then draw the text and a bounding box surrounding
# the text region of the input frame
text = "".join([c if ord(c) < aRange else "" for c in text]).strip()
output = orig.copy()
cv2.rectangle(output, (spX, spY), (epX, epY), drawTag, 2)
cv2.putText(output, text, (spX, spY - 20), cv2.FONT_HERSHEY_SIMPLEX, 1.2, drawTag, 3)
# Show the output frame
cv2.imshow(title, output)
Finally, it fetches the potential text region along with the text & then prints on top of the source video. Also, it removed some non-printable characters during this time to avoid any cryptic texts.
readingVideo.py (Main calling script.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Instantiating all the main class
x1 = rtfs.clsReadingTextFromStream()
# Execute all the pass
r1 = x1.processStream(debugInd, var)
if (r1 == 0):
print('Successfully read text from the Live Stream!')
else:
print('Failed to read text from the Live Stream!')
The above lines instantiate the main calling class & then invoke the function to get the desired extracted text from the live streaming video if that is successful.
FOLDER STRUCTURE:
Here is the folder structure that contains all the files & directories in MAC O/S –
You will get the complete codebase in the following Github link.
Unfortunately, I cannot upload the model due to it’s size. I will share on the need basis.
I’ll bring some more exciting topic in the coming days from the Python verse. Please share & subscribe my post & let me know your feedback.
Till then, Happy Avenging! 🙂
Note: All the data & scenario posted here are representational data & scenarios & available over the internet & for educational purpose only. Some of the images (except my photo) that we’ve used are available over the net. We don’t claim the ownership of these images. There is an always room for improvement & especially the prediction quality.
Today, I’ll be explaining the enhancement of one of the previous posts. I know that I’ve shared the fascinating API named prophet-API, which Facebook developed. One can quickly get more accurate predictions with significantly fewer data points. (If you want to know more about that post, please click on the following link.)
However, there is another enhancement on top of that API, which is more accurate. However, one needs to know – when they should consider using it. So, today, we’ll be talking about the neural prophet API.
But, before we start digging deep, why don’t we view the demo first?
Demo
Let’s visit a diagram. That way, you can understand where you can use it. Also, I’ll be sharing some of the links from the original site for better information mining.
Source: Neural Prophet (Official Site)
As one can see, this API is trying to bridge between the different groups & it enables the time-series computation efficiently.
WHERE TO USE:
Let’s visit another diagram from the same source.
Source: Neural Prophet (Official Site)
So, I hope these two pictures give you a clear picture & relatively set your expectations to more ground reality.
ARCHITECTURE:
Let us explore the architecture –
Architecture Diagram
As one can see, the application is processing IoT data & creating a historical data volume, out of which the model is gradually predicting correct outcomes with higher confidence.
For more information on this API, please visit the following link.
CODE:
Let’s explore the essential scripts here.
clsConfig.py (Configuration file for the entire application.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
As one can see, our application is equipped with the events to predict our use case better.
2. clsPredictIonIoT.py (Main class file, which will invoke neural-prophet forecast for the entire application.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Some of the key snippets that I will discuss here are as follows –
## user specified events
# history events
SummerEnd = p.DataFrame(event1)
LongWeekend = p.DataFrame(event2)
dfEvents = p.concat((SummerEnd, LongWeekend))
# NeuralProphet Object
# Adding events
m = NeuralProphet(loss_func="MSE")
# set the model to expect these events
m = m.add_events(["SummerEnd", "LongWeekend"])
# create the data df with events
historyDf = m.create_df_with_events(df, dfEvents)
Creating & adding events into your model will allow it to predict based on the milestones.
# fit the model
metrics = m.fit(historyDf, freq="D")
# forecast with events known ahead
futureDf = m.make_future_dataframe(df=historyDf, events_df=dfEvents, periods=365, n_historic_predictions=len(df))
forecastDf = m.predict(df=futureDf)
events = forecastDf[(forecastDf['event_SummerEnd'].abs() + forecastDf['event_LongWeekend'].abs()) > 0]
events.tail()
## plotting forecasts
fig = m.plot(forecastDf)
## plotting components
figComp = m.plot_components(forecastDf)
## plotting parameters
figParam = m.plot_parameters()
Based on the daily/monthly collected data, our algorithm tries to plot the data points & predict a future trend, which will look like this –
Future Data Points
From the above diagram, we can conclude that the CPU’s trend has been growing day by day since the beginning. However, there are some events when we can see a momentary drop in requirements due to the climate & holidays. During those times, either people are not using them or are not at home.
Apart from that, I’ve demonstrated the use of a benchwork framework, & splitting the data into Train, Test & Validation & captured the RMSE values. I would request you to go through that & post any questions if you have any.
You can witness the train & validation datasets & visualize them in the standard manner, which will look something like –
Demo
3. readingIoT.py (Main invoking script.)
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
# Reading the source IoT data
iotData = p.read_csv(fileName)
df = iotData.rename(columns={'MonthlyDate': 'ds', 'AvgIoTCPUUsage': 'y'})[['ds', 'y']]
r1 = x1.forecastSeries(df)
if (r1 == 0):
print('Successfully IoT forecast predicted!')
else:
print('Failed to predict IoT forecast!')
var2 = datetime.datetime.now()
In those above lines, the main calling application is invoking the neural-forecasting class & passing the pandas dataframe containing IoT’s historical data to train its model.
For your information, here is the outcome of the run, when you invoke the main calling script –
Demo – Continue
FOLDER STRUCTURE:
Please find the folder structure as shown –
Directory Structure
So, we’ve done it.
You will get the complete codebase in the following Github link.
I’ll bring some more exciting topic in the coming days from the Python verse. Please share & subscribe my post & let me know your feedback.
Till then, Happy Avenging! 😀
Note: All the data & scenario posted here are representational data & scenarios & available over the internet & for educational purpose only. Some of the images (except my photo) that we’ve used are available over the net. We don’t claim the ownership of these images. There is an always room for improvement & especially the prediction quality.
You must be logged in to post a comment.