In this post, I’ll be showing how to create a popular app by taking screenshots & right bits of prompt engineering by using Python & react.
Please wait for the post!
This site mainly deals with various use cases demonstrated using Python, Data Science, Cloud basics, SQL Server, Oracle, Teradata along with SQL & their implementation. Expecting yours active participation & time. This blog can be access from your TP, Tablet & mobile also. Please provide your feedback.
In this post, I’ll be showing how to create a popular app by taking screenshots & right bits of prompt engineering by using Python & react.
Please wait for the post!
Introduction
At the recent Argyle AI Summit, a prestigious event in the AI industry, I had the honor of participating as a speaker alongside esteemed professionals like Misha Leybovich from Google Labs. The summit, coordinated by Sylvia Das Chagas, a former senior AI conversation designer at CVS Health, provided an enlightening platform to discuss the evolving role of AI in talent management. Our session focused on the theme “Driving Talent with AI,” addressing some of the most pressing questions in the field. Frequently, relevant use cases were shared in detail to support these threads.
To view the actual page, please click the following link.
Impact of AI on Talent Management
One of the critical topics we explored was AI’s impact on talent management in the upcoming year. AI’s influence in hiring and retention is becoming increasingly significant. For example, AI-powered tools can now analyze vast amounts of data to identify the best candidates for a role, going beyond traditional resume screening. In retention, AI is instrumental in identifying patterns that indicate an employee’s likelihood to leave, enabling proactive measures.
Dispelling Fears Around AI Replacing Jobs
A burning question in AI is how leaders address fears that AI might replace manual jobs. We discussed the importance of leaders framing AI as a complement to human skills rather than a replacement. AI enhances employee capabilities by automating mundane tasks, allowing employees to focus on more creative and strategic work.
Innovative AI Tools for Organizations
Regarding new AI tools that organizations should watch out for, the conversation highlighted tools that enhance remote collaboration and workplace inclusivity. Tools like virtual meeting assistants that can transcribe, translate, and summarize meetings in real time are becoming invaluable in today’s global work environment.
AI in Boosting Employee Motivation and Productivity
AI’s role in boosting employee motivation and productivity was another focal point. We discussed how AI-driven career development programs can offer personalized learning paths, helping employees grow and stay motivated.
Incorporating Multilingual Capabilities in AI Tools
Incorporating multiple languages in tools like ChatGPT was highlighted as a critical step towards inclusivity. This expansion allows a broader range of employees to interact with AI tools in their native language, fostering a more inclusive workplace environment.
Addressing Reluctance to Change
Lastly, we tackled the challenge of addressing employees’ reluctance to change. Emphasizing the importance of transparent communication and education about AI’s benefits was identified as key. Organizations can alleviate fears and encourage a more accepting attitude towards AI by involving employees in the AI implementation process and providing training.
Conclusion
The Argyle AI Summit offered a compelling glimpse into the future of AI in talent management. The session provided valuable insights for leaders looking to harness AI’s potential to enhance talent management strategies by discussing real-world examples and strategies. To gain more in-depth knowledge and perspectives shared during this summit, I encourage interested parties to visit the recorded session link for a more comprehensive understanding.
Or, you can directly view it from here –
Feedback Request
I would greatly appreciate your feedback on the insights shared during the summit. Your thoughts and perspectives are invaluable as we continue to explore and navigate the evolving landscape of AI in the workplace.
Note: Video content hosted at a third-party site by the summit organizer & not by me.
Today, I will share a new post that will contextualize the source files & then read the data into the pandas data frame, and then dynamically create the SQL & execute it. Then, fetch the data from the sources based on the query generated dynamically. This project is for the advanced Python developer and data Science Newbie.
In this post, I’ve directly subscribed to OpenAI & I’m not using OpenAI from Azure. However, I’ll explore that in the future as well.
Before I explain the process to invoke this new library, why not view the demo first & then discuss it?
FLOW OF EVENTS:
Let us look at the flow diagram as it captures the sequence of events that unfold as part of the process.
The application will take the metadata captured from source data dynamically. It blends the metadata and enhances the prompt to pass to the Flask server. The Flask server has all the limits of contexts.
Once the application receives the correct generated SQL, it will then apply the SQL using the SQLAlchemy package to get the desired results.
IMPORTANT PACKAGES:
The following are the important packages that are essential to this project –
pip install openai==1.6.1
pip install pandas==2.1.4
pip install Flask==3.0.0
pip install SQLAlchemy==2.0.23
CODE:
We’ll have both the server and the main application. Today, we’ll be going in reverse mode. We first discuss the main script & then explain all the other class scripts.
Please find some of the key snippet from this discussion –
@app.route('/message', methods=['POST'])
def message():
input_text = request.json.get('input_text', None)
session_id = request.json.get('session_id', None)
print('*' * 240)
print('User Input:')
print(str(input_text))
print('*' * 240)
# Retrieve conversation history from the session or database
conversation_history = session.get(session_id, [])
# Add the new message to the conversation history
conversation_history.append(input_text)
# Call OpenAI API with the updated conversation
response = client.with_options(max_retries=0).chat.completions.create(
messages=[
{
"role": "user",
"content": input_text,
}
],
model=cf.conf['MODEL_NAME'],
)
# Extract the content from the first choice's message
chat_response = response.choices[0].message.content
print('*' * 240)
print('Resposne::')
print(chat_response)
print('*' * 240)
conversation_history.append(chat_response)
# Store the updated conversation history in the session or database
session[session_id] = conversation_history
return chat_response
This code defines a web application route that handles POST requests sent to the /message
endpoint:
@app.route('/message', methods=['POST'])
part specifies that the function message()
is executed when the server receives a POST request at the /message
URL.message()
function:
input_text
(the user’s input message) and session_id
(a unique identifier for the user’s session).session_id
. This history is a list of messages.input_text
) to this conversation history.max_retries=0
).cf.conf['MODEL_NAME']
).session_id
.Now, let us understand the few important piece of snippet –
def text2SQLBegin(self, DBFileNameList, fileDBPath, srcQueryPrompt, joinCond, debugInd='N'):
question = srcQueryPrompt
create_table_statement = ''
jStr = ''
print('DBFileNameList::', DBFileNameList)
print('prevSessionDBFileNameList::', self.prevSessionDBFileNameList)
if set(self.prevSessionDBFileNameList) == set(DBFileNameList):
self.flag = 'Y'
else:
self.flag = 'N'
if self.flag == 'N':
for i in DBFileNameList:
DBFileName = i
FullDBname = fileDBPath + DBFileName
print('File: ', str(FullDBname))
tabName, _ = DBFileName.split('.')
# Reading the source data
df = pd.read_csv(FullDBname)
# Convert all string columns to lowercase
df = df.apply(lambda x: x.str.lower() if x.dtype == "object" else x)
# Convert DataFrame to SQL table
df.to_sql(tabName, con=engine, index=False)
# Create a MetaData object and reflect the existing database
metadata = MetaData()
metadata.reflect(bind=engine)
# Access the 'users' table from the reflected metadata
table = metadata.tables[tabName]
# Generate the CREATE TABLE statement
create_table_statement = create_table_statement + str(CreateTable(table)) + '; \n'
tabName = ''
for joinS in joinCond:
jStr = jStr + joinS + '\n'
self.prevSessionDBFileNameList = DBFileNameList
self.prev_create_table_statement = create_table_statement
masterSessionDBFileNameList = self.prevSessionDBFileNameList
mast_create_table_statement = self.prev_create_table_statement
else:
masterSessionDBFileNameList = self.prevSessionDBFileNameList
mast_create_table_statement = self.prev_create_table_statement
inputPrompt = (templateVal_1 + mast_create_table_statement + jStr + templateVal_2).format(question=question)
if debugInd == 'Y':
print('INPUT PROMPT::')
print(inputPrompt)
print('*' * 240)
print('Find the Generated SQL:')
print()
DBFileNameList = []
create_table_statement = ''
return inputPrompt
text2SQLBegin
function processes a list of database file names (DBFileNameList
), a file path (fileDBPath
), a query prompt (srcQueryPrompt
), join conditions (joinCond
), and a debug indicator (debugInd
) to generate SQL commands.pandas
library.CREATE TABLE
SQL statement is created.DBFileNameList
and create_table_statement
variables, and returns the constructed input prompt. def text2SQLEnd(self, srcContext, debugInd='N'):
url = self.url
payload = json.dumps({"input_text": srcContext,"session_id": ""})
headers = {'Content-Type': 'application/json', 'Cookie': cf.conf['HEADER_TOKEN']}
response = requests.request("POST", url, headers=headers, data=payload)
return response.text
The text2SQLEnd
function sends an HTTP POST request to a specified URL and returns the response. It takes two parameters: srcContext
which contains the input text, and an optional debugInd
for debugging purposes. The function constructs the request payload by converting the input text and an empty session ID to JSON format. It sets the request headers, including a content type of ‘application/json’ and a token from the configuration file. The function then sends the POST request using the requests
library and returns the text content of the response.
def sql2Data(self, srcSQL):
# Executing the query on top of your data
resultSQL = pd.read_sql_query(srcSQL, con=engine)
return resultSQL
The sql2Data
function is designed to execute a SQL query on a database and return the result. It takes a single parameter, srcSQL
, which contains the SQL query to be executed. The function uses the pandas
library to run the provided SQL query (srcSQL
) against a database connection (engine
). It then returns the result of this query, which is typically a DataFrame object containing the data retrieved from the database.
def genData(self, srcQueryPrompt, fileDBPath, DBFileNameList, joinCond, debugInd='N'):
try:
authorName = self.authorName
website = self.website
var = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
print('*' * 240)
print('SQL Start Time: ' + str(var))
print('*' * 240)
print('*' * 240)
print()
if debugInd == 'Y':
print('Author Name: ', authorName)
print('For more information, please visit the following Website: ', website)
print()
print('*' * 240)
print('Your Data for Retrieval:')
print('*' * 240)
if debugInd == 'Y':
print()
print('Converted File to Dataframe Sample:')
print()
else:
print()
context = self.text2SQLBegin(DBFileNameList, fileDBPath, srcQueryPrompt, joinCond, debugInd)
srcSQL = self.text2SQLEnd(context, debugInd)
print(srcSQL)
print('*' * 240)
print()
resDF = self.sql2Data(srcSQL)
print('*' * 240)
print('SQL End Time: ' + str(var))
print('*' * 240)
return resDF
except Exception as e:
x = str(e)
print('Error: ', x)
df = pd.DataFrame()
return df
authorName
, website
, and a timestamp (var
). It then prints the start time of the SQL process. If the debug indicator (debugInd
) is ‘Y’, it prints additional information like the author’s name and website.text2SQLBegin
with various parameters (file paths, database file names, query prompt, join conditions, and the debug indicator) to generate an SQL context. Then it calls text2SQLEnd
with this context and the debug indicator to generate the actual SQL query.sql2Data
, which returns the result as a data frame (resDF
).resDF
) containing the results of the executed SQL query. If an error occurs, it returns an empty DataFrame instead.DIRECTORY STRUCTURES:
Let us explore the directory structure starting from the parent to some of the important child folder should look like this –
Let us understand the important screenshots of this entire process –
So, finally, we’ve done it.
You will get the complete codebase in the following GitHub link.
I’ll bring some more exciting topics in the coming days from the Python verse. Please share & subscribe to my post & let me know your feedback.
Till then, Happy Avenging! 🙂
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. Some of the images (except my photo) we’ve used are available over the net. We don’t claim ownership of these images. There is always room for improvement & especially in the prediction quality.
Today, I’ll be presenting another exciting capability of architecture in the world of LLMs, where you need to answer one crucial point & that is how valid the response generated by these LLMs is against your data. This response is critical when discussing business growth & need to take the right action at the right time.
Why not view the demo before going through it?
Isn’t it exciting? Great! Let us understand this in detail.
Flow of Architecture:
The first dotted box (extreme-left) represents the area that talks about the data ingestion from different sources, including third-party PDFs. It is expected that organizations should have ready-to-digest data sources. Examples: Data Lake, Data Mart, One Lake, or any other equivalent platforms. Those PDFs will provide additional insights beyond the conventional advanced analytics.
You need to have some kind of OCR solution that will extract all the relevant information in the form of text from the documents.
The next important part is how you define the chunking & embedding of data chunks into Vector DB. Chunking & indexing strategies, along with the overlapping chain, play a crucial importance in tying that segregated piece of context into a single context that will be fed into the source for your preferred LLMs.
This system employs a vector similarity search to browse through unstructured information and concurrently accesses the database to retrieve the context, ensuring that the responses are not only comprehensive but also anchored in validated knowledge.
This approach is particularly vital for addressing multi-hop questions, where a single query can be broken down into multiple sub-questions and may require information from numerous documents to generate an accurate answer.
Python Packages:
pip install openai==0.27.8
pip install pandas==2.0.3
pip install tensorflow==2.11.1
pip install faiss-cpu==1.7.4
pip install gensim==4.3.2
Let us understand the key class & snippets.
Let us understand some of the key snippets from the above script (Full scripts will be available in the GitHub Repo) –
# Sample function to convert text to a vector
def text2Vector(self, text):
# Encode the text using the tokenizer
words = [word for word in text.lower().split() if word in self.model]
# If no words in the model, return a zero vector
if not words:
return np.zeros(self.model.vector_size)
# Compute the average of the word vectors
vector = np.mean([self.model[word] for word in words], axis=0)
return vector.reshape(1, -1)
This code is for a function called “text2Vector” that takes some text as input and converts it into a numerical vector. Let me break it down step by step:
So, in simple terms, this function takes a piece of text, looks up the word vectors for the words in that text, and calculates the average of those vectors to create a single numerical representation of the text. If none of the words are found in the model, it returns a vector of zeros.
def genData(self):
try:
basePath = self.basePath
modelFileName = self.modelFileName
vectorDBPath = self.vectorDBPath
vectorDBFileName = self.vectorDBFileName
# Create a FAISS index
dimension = int(cf.conf['NO_OF_MODEL_DIM']) # Assuming 100-dimensional vectors
index = faiss.IndexFlatL2(dimension)
print('*' * 240)
print('Vector Index Your Data for Retrieval:')
print('*' * 240)
FullVectorDBname = vectorDBPath + vectorDBFileName
indexFile = str(vectorDBPath) + str(vectorDBFileName) + '.index'
print('File: ', str(indexFile))
data = {}
# List all files in the specified directory
files = os.listdir(basePath)
# Filter out files that are not text files
text_files = [file for file in files if file.endswith('.txt')]
# Read each text file
for file in text_files:
file_path = os.path.join(basePath, file)
print('*' * 240)
print('Processing File:')
print(str(file_path))
try:
# Attempt to open with utf-8 encoding
with open(file_path, 'r', encoding='utf-8') as file:
for line_number, line in enumerate(file, start=1):
# Assume each line is a separate document
vector = self.text2Vector(line)
vector = vector.reshape(-1)
index_id = index.ntotal
index.add(np.array([vector])) # Adding the vector to the index
data[index_id] = {'text': line, 'line_number': line_number, 'file_name': file_path} # Storing the line and file name
except UnicodeDecodeError:
# If utf-8 fails, try a different encoding
try:
with open(file_path, 'r', encoding='ISO-8859-1') as file:
for line_number, line in enumerate(file, start=1):
# Assume each line is a separate document
vector = self.text2Vector(line)
vector = vector.reshape(-1)
index_id = index.ntotal
index.add(np.array([vector])) # Adding the vector to the index
data[index_id] = {'text': line, 'line_number': line_number, 'file_name': file_path} # Storing the line and file name
except Exception as e:
print(f"Could not read file {file}: {e}")
continue
print('*' * 240)
# Save the data dictionary using pickle
dataCache = vectorDBPath + modelFileName
with open(dataCache, 'wb') as f:
pickle.dump(data, f)
# Save the index and data for later use
faiss.write_index(index, indexFile)
print('*' * 240)
return 0
except Exception as e:
x = str(e)
print('Error: ', x)
return 1
In summary, this function reads text files, converts their contents into numerical vectors, and builds a FAISS index for efficient similarity search. It also saves the processed data and the index for later use. If there are any issues during the process, it prints error messages but continues processing other files.
Let us understand some of the key snippets from the above script (Full scripts will be available in the GitHub Repo) –
def ragAnswerWithHaystackAndGPT3(self, queryVector, k, question):
modelName = self.modelName
maxToken = self.maxToken
temp = self.temp
# Assuming getTopKContexts is a method that returns the top K contexts
contexts = self.getTopKContexts(queryVector, k)
messages = []
# Add contexts as system messages
for file_name, line_number, text in contexts:
messages.append({"role": "system", "content": f"Document: {file_name} \nLine Number: {line_number} \nContent: {text}"})
prompt = self.generateOpenaiPrompt(queryVector, k)
prompt = prompt + "Question: " + str(question) + ". \n Answer based on the above documents."
# Add user question
messages.append({"role": "user", "content": prompt})
# Create chat completion
completion = client.chat.completions.create(
model=modelName,
messages=messages,
temperature = temp,
max_tokens = maxToken
)
# Assuming the last message in the response is the answer
last_response = completion.choices[0].message.content
source_refernces = ['FileName: ' + str(context[0]) + ' - Line Numbers: ' + str(context[1]) + ' - Source Text (Reference): ' + str(context[2]) for context in contexts]
return last_response, source_refernces
In summary, this function takes a user’s query, retrieves relevant contexts or documents, sets up a conversation with GPT-3 that includes the query and contexts, and then uses GPT-3 to generate an answer. It also provides references to the sources used in generating the answer.
def getTopKContexts(self, queryVector, k):
try:
distances, indices = index.search(queryVector, k)
resDict = [(data[i]['file_name'], data[i]['line_number'], data[i]['text']) for i in indices[0]]
return resDict
except Exception as e:
x = str(e)
print('Error: ', x)
return x
This code defines a function called getTopKContexts
. Its purpose is to retrieve the top K relevant contexts or pieces of information from a pre-built index based on a query vector. Here’s a breakdown of what it does:
queryVector
, which is a numerical vector representing a query, and k
, which specifies how many relevant contexts to retrieve.index.search
method to find the top K closest contexts to the given queryVector
. This method returns two arrays: distances
(measuring how similar the contexts are to the query) and indices
(indicating the positions of the closest contexts in the data).resDict"
, which contains tuples for each of the top K contexts. Each tuple contains three pieces of information: the file name (file_name
), the line number (line_number
), and the text content (text
) of the context. These details are extracted from a data dictionary.resDict
) to the caller.x
), prints the error message, and then returns the error message itself.In summary, this function takes a query vector and finds the K most relevant contexts or pieces of information based on their similarity to the query. It returns these contexts as a list of tuples containing file names, line numbers, and text content. If there’s an error, it prints an error message and returns the error message string.
def generateOpenaiPrompt(self, queryVector, k):
contexts = self.getTopKContexts(queryVector, k)
template = ct.templateVal_1
prompt = template
for file_name, line_number, text in contexts:
prompt += f"Document: {file_name}\n Line Number: {line_number} \n Content: {text}\n\n"
return prompt
This code defines a function called generateOpenaiPrompt
. Its purpose is to create a prompt or a piece of text that combines a template with information from the top K relevant contexts retrieved earlier. Let’s break down what it does:
getTopKContexts
function to obtain the top K relevant contexts based on a given queryVector
.template
with a predefined template value (likely defined elsewhere in the code).prompt
variable to the initial template
.prompt
. Specifically, it adds lines to the prompt
that include:
Document: [file_name]
).Line Number: [line_number]
).Content: [text]
).prompt
, which is a combination of the template and information from the relevant contexts.In summary, this function takes a query vector, retrieves relevant contexts, and creates a prompt by combining a template with information from these contexts. This prompt can then be used as input for an AI model or system, likely for generating responses or answers based on the provided context.
Let us understand the directory structure of this entire application –
To learn more about this package, please visit the following GitHub link.
So, finally, we’ve done it. I know that this post is relatively smaller than my earlier post. But, I think, you can get a good hack to improve some of your long-running jobs by applying this trick.
I’ll bring some more exciting topics in the coming days from the Python verse. Please share & subscribe to my post & let me know your feedback.
Till then, Happy Avenging! 🙂
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. Some of the images (except my photo) we’ve used are available over the net. We don’t claim ownership of these images. There is always room for improvement & especially in the prediction quality.
Today, we’ll share the second installment of the RAG implementation. If you are new here, please visit the previous post for full context.
In this post, we’ll be discussing the Haystack framework more. Again, before discussing the main context, I want to present the demo here.
FLOW OF EVENTS:
Let us look at the flow diagram as it captures the sequence of events that unfold as part of the process, where today, we’ll pay our primary attention.
As you can see today, we’ll discuss the red dotted line, which contextualizes the source data into the Vector DBs.
Let us understand the flow of events here –
IMPORTANT PACKAGES:
pip install farm-haystack==1.19.0
pip install Flask==2.2.5
pip install Flask-Cors==4.0.0
pip install Flask-JWT-Extended==4.5.2
pip install Flask-Session==0.5.0
pip install openai==0.27.8
pip install pandas==2.0.3
pip install tensorflow==2.11.1
We’re using the Metropolitan Museum API to feed the data to our Vector DB. For more information, please visit the following link. And this is free to use & moreover, we’re using it for education scenarios.
CODE:
We’ll discuss the tokenization part highlighted in a red dotted line from the above picture.
Python:
We’ll discuss the scripts in the diagram as part of the flow mentioned above.
def genData(self):
try:
base_url = self.base_url
header_token = self.header_token
basePath = self.basePath
outputPath = self.outputPath
mergedFile = self.mergedFile
subdir = self.subdir
Ind = self.Ind
var_1 = datetime.now().strftime("%H.%M.%S")
devVal = list()
objVal = list()
# Main Details
headers = {'Cookie':header_token}
payload={}
url = base_url + '/departments'
date_ranges = self.generateFirstDayOfLastTenYears()
# Getting all the departments
try:
print('Department URL:')
print(str(url))
response = requests.request("GET", url, headers=headers, data=payload)
parsed_data = json.loads(response.text)
print('Department JSON:')
print(str(parsed_data))
# Extract the "departmentId" values into a Python list
for dept_det in parsed_data['departments']:
for info in dept_det:
if info == 'departmentId':
devVal.append(dept_det[info])
except Exception as e:
x = str(e)
print('Error: ', x)
devVal = list()
# List to hold thread objects
threads = []
# Calling the Data using threads
for dep in devVal:
t = threading.Thread(target=self.getDataThread, args=(dep, base_url, headers, payload, date_ranges, objVal, subdir, Ind,))
threads.append(t)
t.start()
# Wait for all threads to complete
for t in threads:
t.join()
res = self.mergeCsvFilesInDirectory(basePath, outputPath, mergedFile)
if res == 0:
print('Successful!')
else:
print('Failure!')
return 0
except Exception as e:
x = str(e)
print('Error: ', x)
return 1
The above code translates into the following steps –
def generateFirstDayOfLastTenYears(self):
yearRange = self.yearRange
date_format = "%Y-%m-%d"
current_year = datetime.now().year
date_ranges = []
for year in range(current_year - yearRange, current_year + 1):
first_day_of_year_full = datetime(year, 1, 1)
first_day_of_year = first_day_of_year_full.strftime(date_format)
date_ranges.append(first_day_of_year)
return date_ranges
The first method will generate the first day of each year for the last ten years, including the current year.
def getDataThread(self, dep, base_url, headers, payload, date_ranges, objVal, subdir, Ind):
try:
cnt = 0
cnt_x = 1
var_1 = datetime.now().strftime("%H.%M.%S")
for x_start_date in date_ranges:
try:
urlM = base_url + '/objects?metadataDate=' + str(x_start_date) + '&departmentIds=' + str(dep)
print('Nested URL:')
print(str(urlM))
response_obj = requests.request("GET", urlM, headers=headers, data=payload)
objectDets = json.loads(response_obj.text)
for obj_det in objectDets['objectIDs']:
objVal.append(obj_det)
for objId in objVal:
urlS = base_url + '/objects/' + str(objId)
print('Final URL:')
print(str(urlS))
response_det = requests.request("GET", urlS, headers=headers, data=payload)
objDetJSON = response_det.text
retDB = self.createData(objDetJSON)
retDB['departmentId'] = str(dep)
if cnt == 0:
df_M = retDB
else:
d_frames = [df_M, retDB]
df_M = pd.concat(d_frames)
if cnt == 1000:
cnt = 0
clog.logr('df_M_' + var_1 + '_' + str(cnt_x) + '_' + str(dep) +'.csv', Ind, df_M, subdir)
cnt_x += 1
df_M = pd.DataFrame()
cnt += 1
except Exception as e:
x = str(e)
print('Error X:', x)
return 0
except Exception as e:
x = str(e)
print('Error: ', x)
return 1
The above method will invoke the individual API call to fetch the relevant artifact information.
def mergeCsvFilesInDirectory(self, directory_path, output_path, output_file):
try:
csv_files = [file for file in os.listdir(directory_path) if file.endswith('.csv')]
data_frames = []
for file in csv_files:
encodings_to_try = ['utf-8', 'utf-8-sig', 'latin-1', 'cp1252']
for encoding in encodings_to_try:
try:
FullFileName = directory_path + file
print('File Name: ', FullFileName)
df = pd.read_csv(FullFileName, encoding=encoding)
data_frames.append(df)
break # Stop trying other encodings if the reading is successful
except UnicodeDecodeError:
continue
if not data_frames:
raise Exception("Unable to read CSV files. Check encoding or file format.")
merged_df = pd.concat(data_frames, ignore_index=True)
merged_full_name = os.path.join(output_path, output_file)
merged_df.to_csv(merged_full_name, index=False)
for file in csv_files:
os.remove(os.path.join(directory_path, file))
return 0
except Exception as e:
x = str(e)
print('Error: ', x)
return 1
The above method will merge all the small files into a single, more extensive historical data that contains over ten years of data (the first day of ten years of data, to be precise).
For the complete code, please visit the GitHub.
#########################################################
#### Written By: SATYAKI DE ####
#### Written On: 27-Jun-2023 ####
#### Modified On 28-Jun-2023 ####
#### ####
#### Objective: This is the main calling ####
#### python script that will invoke the ####
#### shortcut application created inside MAC ####
#### enviornment including MacBook, IPad or IPhone. ####
#### ####
#########################################################
import datetime
from clsConfigClient import clsConfigClient as cf
import clsExtractJSON as cej
########################################################
################ Global Area ######################
########################################################
cJSON = cej.clsExtractJSON()
basePath = cf.conf['DATA_PATH']
outputPath = cf.conf['OUTPUT_PATH']
mergedFile = cf.conf['MERGED_FILE']
########################################################
################ End Of Global Area #################
########################################################
# Disbling Warning
def warn(*args, **kwargs):
pass
import warnings
warnings.warn = warn
def main():
try:
var = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
print('*'*120)
print('Start Time: ' + str(var))
print('*'*120)
r1 = cJSON.genData()
if r1 == 0:
print()
print('Successfully Scrapped!')
else:
print()
print('Failed to Scrappe!')
print('*'*120)
var1 = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
print('End Time: ' + str(var1))
except Exception as e:
x = str(e)
print('Error: ', x)
if __name__ == '__main__':
main()
The above script calls the main class after instantiating the class.
def createRec(self):
try:
basePath = self.basePath
fileName = self.fileName
Ind = self.Ind
subdir = self.subdir
base_url = self.base_url
outputPath = self.outputPath
mergedFile = self.mergedFile
cleanedFile = self.cleanedFile
FullFileName = outputPath + mergedFile
df = pd.read_csv(FullFileName)
df2 = df[listCol]
dfFin = df2.drop_duplicates().reset_index(drop=True)
dfFin['artist_URL'] = dfFin['artistWikidata_URL'].combine_first(dfFin['artistULAN_URL'])
dfFin['object_URL'] = dfFin['objectURL'].combine_first(dfFin['objectWikidata_URL'])
dfFin['Wiki_URL'] = dfFin['Wikidata_URL'].combine_first(dfFin['AAT_URL']).combine_first(dfFin['URL']).combine_first(dfFin['object_URL'])
# Dropping the old Dtype Columns
dfFin.drop(['artistWikidata_URL'], axis=1, inplace=True)
dfFin.drop(['artistULAN_URL'], axis=1, inplace=True)
dfFin.drop(['objectURL'], axis=1, inplace=True)
dfFin.drop(['objectWikidata_URL'], axis=1, inplace=True)
dfFin.drop(['AAT_URL'], axis=1, inplace=True)
dfFin.drop(['Wikidata_URL'], axis=1, inplace=True)
dfFin.drop(['URL'], axis=1, inplace=True)
# Save the filtered DataFrame to a new CSV file
#clog.logr(cleanedFile, Ind, dfFin, subdir)
res = self.addHash(dfFin)
if res == 0:
print('Added Hash!')
else:
print('Failed to add hash!')
# Generate the text for each row in the dataframe
for _, row in dfFin.iterrows():
x = self.genPrompt(row)
self.addDocument(x, cleanedFile)
return documents
except Exception as e:
x = str(e)
print('Record Error: ', x)
return documents
The above code will read the data from the extensive historical file created from the earlier steps & then it will clean the file by removing all the duplicate records (if any) & finally, it will create three unique URLs that constitute artist, object & wiki.
Also, this application will remove the hyperlink with a specific hash value, which will feed into the vector DB. Vector DB could be better with the URLs. Hence, we will store the URLs in a separate file by storing the associate hash value & later, we’ll fetch it in a lookup from the open AI response.
Then, this application will generate prompts dynamically & finally create the documents for later steps of vector DB consumption by invoking the addDocument() methods.
For more details, please visit the GitHub link.
#########################################################
#### Written By: SATYAKI DE ####
#### Written On: 27-Jun-2023 ####
#### Modified On 28-Jun-2023 ####
#### ####
#### Objective: This is the main calling ####
#### python script that will invoke the ####
#### shortcut application created inside MAC ####
#### enviornment including MacBook, IPad or IPhone. ####
#### ####
#########################################################
from clsConfigClient import clsConfigClient as cf
import clsL as log
import clsCreateList as ccl
from datetime import datetime, timedelta
# Disbling Warning
def warn(*args, **kwargs):
pass
import warnings
warnings.warn = warn
###############################################
### Global Section ###
###############################################
#Initiating Logging Instances
clog = log.clsL()
cl = ccl.clsCreateList()
var = datetime.now().strftime(".%H.%M.%S")
documents = []
###############################################
### End of Global Section ###
###############################################
def main():
try:
var = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
print('*'*120)
print('Start Time: ' + str(var))
print('*'*120)
print('*'*240)
print('Creating Index store:: ')
print('*'*240)
documents = cl.createRec()
print('Inserted Sample Records: ')
print(str(documents))
print('\n')
r1 = len(documents)
if r1 > 0:
print()
print('Successfully Indexed sample records!')
else:
print()
print('Failed to sample Indexed recrods!')
print('*'*120)
var1 = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
print('End Time: ' + str(var1))
except Exception as e:
x = str(e)
print('Error: ', x)
if __name__ == '__main__':
main()
The above script invokes the main class after instantiating it & invokes the createRec() methods to tokenize the data into the vector DB.
#########################################################
#### Written By: SATYAKI DE ####
#### Written On: 27-Jun-2023 ####
#### Modified On 28-Sep-2023 ####
#### ####
#### Objective: This is the main calling ####
#### python script that will invoke the ####
#### haystack frameowrk to contextulioze the docs ####
#### inside the vector DB. ####
#### ####
#########################################################
from haystack.document_stores.faiss import FAISSDocumentStore
from haystack.nodes import DensePassageRetriever
import openai
import pandas as pd
import os
import clsCreateList as ccl
from clsConfigClient import clsConfigClient as cf
import clsL as log
from datetime import datetime, timedelta
# Disbling Warning
def warn(*args, **kwargs):
pass
import warnings
warnings.warn = warn
###############################################
### Global Section ###
###############################################
Ind = cf.conf['DEBUG_IND']
openAIKey = cf.conf['OPEN_AI_KEY']
os.environ["TOKENIZERS_PARALLELISM"] = "false"
#Initiating Logging Instances
clog = log.clsL()
cl = ccl.clsCreateList()
var = datetime.now().strftime(".%H.%M.%S")
# Encode your data to create embeddings
documents = []
var_1 = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
print('*'*120)
print('Start Time: ' + str(var_1))
print('*'*120)
print('*'*240)
print('Creating Index store:: ')
print('*'*240)
documents = cl.createRec()
print('Inserted Sample Records: ')
print(documents[:5])
print('\n')
print('Type:')
print(type(documents))
r1 = len(documents)
if r1 > 0:
print()
print('Successfully Indexed records!')
else:
print()
print('Failed to Indexed recrods!')
print('*'*120)
var_2 = datetime.now().strftime("%Y-%m-%d_%H-%M-%S")
print('End Time: ' + str(var_2))
# Passing OpenAI API Key
openai.api_key = openAIKey
###############################################
### End of Global Section ###
###############################################
class clsFeedVectorDB:
def __init__(self):
self.basePath = cf.conf['DATA_PATH']
self.modelFileName = cf.conf['CACHE_FILE']
self.vectorDBPath = cf.conf['VECTORDB_PATH']
self.vectorDBFileName = cf.conf['VECTORDB_FILE_NM']
self.queryModel = cf.conf['QUERY_MODEL']
self.passageModel = cf.conf['PASSAGE_MODEL']
def retrieveDocuments(self, question, retriever, top_k=3):
return retriever.retrieve(question, top_k=top_k)
def generateAnswerWithGPT3(self, retrievedDocs, question):
documents_text = " ".join([doc.content for doc in retrievedDocs])
prompt = f"Given the following documents: {documents_text}, answer the question: {question}"
response = openai.Completion.create(
model="text-davinci-003",
prompt=prompt,
max_tokens=150
)
return response.choices[0].text.strip()
def ragAnswerWithHaystackAndGPT3(self, question, retriever):
retrievedDocs = self.retrieveDocuments(question, retriever)
return self.generateAnswerWithGPT3(retrievedDocs, question)
def genData(self, strVal):
try:
basePath = self.basePath
modelFileName = self.modelFileName
vectorDBPath = self.vectorDBPath
vectorDBFileName = self.vectorDBFileName
queryModel = self.queryModel
passageModel = self.passageModel
print('*'*120)
print('Index Your Data for Retrieval:')
print('*'*120)
FullFileName = basePath + modelFileName
FullVectorDBname = vectorDBPath + vectorDBFileName
sqlite_path = "sqlite:///" + FullVectorDBname + '.db'
print('Vector DB Path: ', str(sqlite_path))
indexFile = "vectorDB/" + str(vectorDBFileName) + '.faiss'
indexConfig = "vectorDB/" + str(vectorDBFileName) + ".json"
print('File: ', str(indexFile))
print('Config: ', str(indexConfig))
# Initialize DocumentStore
document_store = FAISSDocumentStore(sql_url=sqlite_path)
libName = "vectorDB/" + str(vectorDBFileName) + '.faiss'
document_store.write_documents(documents)
# Initialize Retriever
retriever = DensePassageRetriever(document_store=document_store,
query_embedding_model=queryModel,
passage_embedding_model=passageModel,
use_gpu=False)
document_store.update_embeddings(retriever=retriever)
document_store.save(index_path=libName, config_path="vectorDB/" + str(vectorDBFileName) + ".json")
print('*'*120)
print('Testing with RAG & OpenAI...')
print('*'*120)
answer = self.ragAnswerWithHaystackAndGPT3(strVal, retriever)
print('*'*120)
print('Testing Answer:: ')
print(answer)
print('*'*120)
return 0
except Exception as e:
x = str(e)
print('Error: ', x)
return 1
In the above script, the following essential steps took place –
Here is a short clip of how the RAG models contextualize with the source data.
So, finally, we’ve done it.
I know that this post is relatively bigger than my earlier post. But, I think, you can get all the details once you go through it.
You will get the complete codebase in the following GitHub link.
I’ll bring some more exciting topics in the coming days from the Python verse. Please share & subscribe to my post & let me know your feedback.
Till then, Happy Avenging! 🙂
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. Some of the images (except my photo) we’ve used are available over the net. We don’t claim ownership of these images. There is always room for improvement & especially in the prediction quality.
Today, I will share a new post in a part series about creating end-end LLMs that feed source data with RAG implementation. I’ll also use OpenAI python-based SDK and Haystack embeddings in this case.
In this post, I’ve directly subscribed to OpenAI & I’m not using OpenAI from Azure. However, I’ll explore that in the future as well.
Before I explain the process to invoke this new library, why not view the demo first & then discuss it?
FLOW OF EVENTS:
Let us look at the flow diagram as it captures the sequence of events that unfold as part of the process.
As you can see, to enable this large & complex solution, we must first establish the capabilities to build applications powered by LLMs, Transformer models, vector search, and more. You can use state-of-the-art NLP models to perform question-answering, answer generation, semantic document search, or build tools capable of complex decision-making and query resolution. Hence, steps no. 1 & 2 showcased the data embedding & creating that informed repository. We’ll be discussing that in our second part.
Once you have the informed repository, the system can interact with the end-users. As part of the query (shown in step 3), the prompt & the question are shared with the process engine, which then turned to reduce the volume & get relevant context from our informed repository & get the tuned context as part of the response (Shown in steps 4, 5 & 6).
Then, this tuned context is shared with the OpenAI for better response & summary & concluding remarks that are very user-friendly & easier to understand for end-users (Shown in steps 8 & 9).
IMPORTANT PACKAGES:
The following are the important packages that are essential to this project –
pip install farm-haystack==1.19.0
pip install Flask==2.2.5
pip install Flask-Cors==4.0.0
pip install Flask-JWT-Extended==4.5.2
pip install Flask-Session==0.5.0
pip install openai==0.27.8
pip install pandas==2.0.3
pip install tensorflow==2.11.1
CODE:
We’ve both the front-end using react & back-end APIs with Python-flask and the Open AI to create this experience.
Python:
Today, we’ll be going in reverse mode. We first discuss the main script & then explain all the other class scripts.
#########################################################
#### Written By: SATYAKI DE ####
#### Written On: 27-Jun-2023 ####
#### Modified On 28-Jun-2023 ####
#### ####
#### Objective: This is the main calling ####
#### python script that will invoke the ####
#### shortcut application created inside MAC ####
#### enviornment including MacBook, IPad or IPhone. ####
#### ####
#########################################################
from flask import Flask, jsonify, request, session
from flask_cors import CORS
from werkzeug.security import check_password_hash, generate_password_hash
from flask_jwt_extended import JWTManager, jwt_required, create_access_token
import pandas as pd
from clsConfigClient import clsConfigClient as cf
import clsL as log
import clsContentScrapper as csc
import clsRAGOpenAI as crao
import csv
from datetime import timedelta
import os
import re
import json
########################################################
################ Global Area ######################
########################################################
#Initiating Logging Instances
clog = log.clsL()
admin_key = cf.conf['ADMIN_KEY']
secret_key = cf.conf['SECRET_KEY']
session_path = cf.conf['SESSION_PATH']
sessionFile = cf.conf['SESSION_CACHE_FILE']
app = Flask(__name__)
CORS(app) # This will enable CORS for all routes
app.config['JWT_SECRET_KEY'] = admin_key # Change this!
app.secret_key = secret_key
jwt = JWTManager(app)
users = cf.conf['USER_NM']
passwd = cf.conf['USER_PWD']
cCScrapper = csc.clsContentScrapper()
cr = crao.clsRAGOpenAI()
# Disbling Warning
def warn(*args, **kwargs):
pass
import warnings
warnings.warn = warn
# Define the aggregation functions
def join_unique(series):
unique_vals = series.drop_duplicates().astype(str)
return ', '.join(filter(lambda x: x != 'nan', unique_vals))
# Building the preaggregate cache
def groupImageWiki():
try:
base_path = cf.conf['OUTPUT_PATH']
inputFile = cf.conf['CLEANED_FILE']
outputFile = cf.conf['CLEANED_FILE_SHORT']
subdir = cf.conf['SUBDIR_OUT']
Ind = cf.conf['DEBUG_IND']
inputCleanedFileLookUp = base_path + inputFile
#Opening the file in dataframe
df = pd.read_csv(inputCleanedFileLookUp)
hash_values = df['Total_Hash'].unique()
dFin = df[['primaryImage','Wiki_URL','Total_Hash']]
# Ensure columns are strings and not NaN
# Convert columns to string and replace 'nan' with an empty string
dFin['primaryImage'] = dFin['primaryImage'].astype(str).replace('nan', '')
dFin['Wiki_URL'] = dFin['Wiki_URL'].astype(str).replace('nan', '')
dFin.drop_duplicates()
# Group by 'Total_Hash' and aggregate
dfAgg = dFin.groupby('Total_Hash').agg({'primaryImage': join_unique,'Wiki_URL': join_unique}).reset_index()
return dfAgg
except Exception as e:
x = str(e)
print('Error: ', x)
df = pd.DataFrame()
return df
resDf = groupImageWiki()
########################################################
################ End Global Area ####################
########################################################
def extractRemoveUrls(hash_value):
image_urls = ''
wiki_urls = ''
# Parse the inner message JSON string
try:
resDf['Total_Hash'] = resDf['Total_Hash'].astype(int)
filtered_df = resDf[resDf['Total_Hash'] == int(hash_value)]
if not filtered_df.empty:
image_urls = filtered_df['primaryImage'].values[0]
wiki_urls = filtered_df['Wiki_URL'].values[0]
return image_urls, wiki_urls
except Exception as e:
x = str(e)
print('extractRemoveUrls Error: ', x)
return image_urls, wiki_urls
def isIncomplete(line):
"""Check if a line appears to be incomplete."""
# Check if the line ends with certain patterns indicating it might be incomplete.
incomplete_patterns = [': [Link](', ': Approximately ', ': ']
return any(line.endswith(pattern) for pattern in incomplete_patterns)
def filterData(data):
"""Return only the complete lines from the data."""
lines = data.split('\n')
complete_lines = [line for line in lines if not isIncomplete(line)]
return '\n'.join(complete_lines)
def updateCounter(sessionFile):
try:
counter = 0
# Check if the CSV file exists
if os.path.exists(sessionFile):
with open(sessionFile, 'r') as f:
reader = csv.reader(f)
for row in reader:
# Assuming the counter is the first value in the CSV
counter = int(row[0])
# Increment counter
counter += 1
# Write counter back to CSV
with open(sessionFile, 'w', newline='') as f:
writer = csv.writer(f)
writer.writerow([counter])
return counter
except Exception as e:
x = str(e)
print('Error: ', x)
return 1
def getPreviousResult():
try:
fullFileName = session_path + sessionFile
newCounterValue = updateCounter(fullFileName)
return newCounterValue
except Exception as e:
x = str(e)
print('Error: ', x)
return 1
@app.route('/login', methods=['POST'])
def login():
username = request.json.get('username', None)
password = request.json.get('password', None)
print('User Name: ', str(username))
print('Password: ', str(password))
#if username not in users or not check_password_hash(users.get(username), password):
if ((username not in users) or (password not in passwd)):
return jsonify({'login': False}), 401
access_token = create_access_token(identity=username)
return jsonify(access_token=access_token)
@app.route('/chat', methods=['POST'])
def get_chat():
try:
#session["key"] = "1D98KI"
#session_id = session.sid
#print('Session Id: ', str(session_id))
cnt = getPreviousResult()
print('Running Session Count: ', str(cnt))
username = request.json.get('username', None)
message = request.json.get('message', None)
print('User: ', str(username))
print('Content: ', str(message))
if cnt == 1:
retList = cCScrapper.extractCatalog()
else:
hashValue, cleanedData = cr.getData(str(message))
print('Main Hash Value:', str(hashValue))
imageUrls, wikiUrls = extractRemoveUrls(hashValue)
print('Image URLs: ', str(imageUrls))
print('Wiki URLs: ', str(wikiUrls))
print('Clean Text:')
print(str(cleanedData))
retList = '{"records":[{"Id":"' + str(cleanedData) + '", "Image":"' + str(imageUrls) + '", "Wiki": "' + str(wikiUrls) + '"}]}'
response = {
'message': retList
}
print('JSON: ', str(response))
return jsonify(response)
except Exception as e:
x = str(e)
response = {
'message': 'Error: ' + x
}
return jsonify(response)
@app.route('/api/data', methods=['GET'])
@jwt_required()
def get_data():
response = {
'message': 'Hello from Flask!'
}
return jsonify(response)
if __name__ == '__main__':
app.run(debug=True)
Let us understand some of the important sections of the above script –
Function – login():
The login function retrieves a ‘username’ and ‘password’ from a JSON request and prints them. It checks if the provided credentials are missing from users or password lists, returning a failure JSON response if so. It creates and returns an access token in a JSON response if valid.
Function – get_chat():
The get_chat function retrieves the running session count and user input from a JSON request. Based on the session count, it extracts catalog data or processes the user’s message from the RAG framework that finally receives the refined response from the OpenAI, extracting hash values, image URLs, and wiki URLs. If an error arises, the function captures and returns the error as a JSON message.
Function – updateCounter():
The updateCounter function checks if a given CSV file exists and retrieves its counter value. It then increments the counter and writes it back to the CSV. If any errors occur, an error message is printed, and the function returns a value of 1.
Function – extractRemoveUrls():
The extractRemoveUrls function attempts to filter a data frame, resDf, based on a provided hash value to extract image and wiki URLs. If the data frame contains matching entries, it retrieves the corresponding URLs. Any errors encountered are printed, but the function always returns the image and wiki URLs, even if they are empty.
#####################################################
#### Written By: SATYAKI DE ####
#### Written On: 27-May-2023 ####
#### Modified On 28-May-2023 ####
#### ####
#### Objective: This is the main calling ####
#### python class that will invoke the ####
#### LangChain of package to extract ####
#### the transcript from the YouTube videos & ####
#### then answer the questions based on the ####
#### topics selected by the users. ####
#### ####
#####################################################
from langchain.document_loaders import YoutubeLoader
from langchain.text_splitter import RecursiveCharacterTextSplitter
from langchain.embeddings.openai import OpenAIEmbeddings
from langchain.vectorstores import FAISS
from langchain.chat_models import ChatOpenAI
from langchain.chains import LLMChain
from langchain.prompts.chat import (
ChatPromptTemplate,
SystemMessagePromptTemplate,
HumanMessagePromptTemplate,
)
from googleapiclient.discovery import build
import clsTemplate as ct
from clsConfigClient import clsConfigClient as cf
import os
from flask import jsonify
import requests
###############################################
### Global Section ###
###############################################
open_ai_Key = cf.conf['OPEN_AI_KEY']
os.environ["OPENAI_API_KEY"] = open_ai_Key
embeddings = OpenAIEmbeddings(openai_api_key=open_ai_Key)
YouTube_Key = cf.conf['YOUTUBE_KEY']
youtube = build('youtube', 'v3', developerKey=YouTube_Key)
# Disbling Warning
def warn(*args, **kwargs):
pass
import warnings
warnings.warn = warn
###############################################
### End of Global Section ###
###############################################
class clsContentScrapper:
def __init__(self):
self.model_name = cf.conf['MODEL_NAME']
self.temp_val = cf.conf['TEMP_VAL']
self.max_cnt = int(cf.conf['MAX_CNT'])
self.url = cf.conf['BASE_URL']
self.header_token = cf.conf['HEADER_TOKEN']
def extractCatalog(self):
try:
base_url = self.url
header_token = self.header_token
url = base_url + '/departments'
print('Full URL: ', str(url))
payload={}
headers = {'Cookie': header_token}
response = requests.request("GET", url, headers=headers, data=payload)
x = response.text
return x
except Exception as e:
discussedTopic = []
x = str(e)
print('Error: ', x)
return x
Let us understand the the core part that require from this class.
Function – extractCatalog():
The extractCatalog function uses specific headers to make a GET request to a constructed URL. The URL is derived by appending ‘/departments’ to a base_url, and a header token is used in the request headers. If successful, it returns the text of the response; if there’s an exception, it prints the error and returns the error message.
#########################################################
#### Written By: SATYAKI DE ####
#### Written On: 27-Jun-2023 ####
#### Modified On 28-Jun-2023 ####
#### ####
#### Objective: This is the main calling ####
#### python script that will invoke the ####
#### shortcut application created inside MAC ####
#### enviornment including MacBook, IPad or IPhone. ####
#### ####
#########################################################
from haystack.document_stores.faiss import FAISSDocumentStore
from haystack.nodes import DensePassageRetriever
import openai
from clsConfigClient import clsConfigClient as cf
import clsL as log
# Disbling Warning
def warn(*args, **kwargs):
pass
import warnings
warnings.warn = warn
import os
import re
###############################################
### Global Section ###
###############################################
Ind = cf.conf['DEBUG_IND']
queryModel = cf.conf['QUERY_MODEL']
passageModel = cf.conf['PASSAGE_MODEL']
#Initiating Logging Instances
clog = log.clsL()
os.environ["TOKENIZERS_PARALLELISM"] = "false"
vectorDBFileName = cf.conf['VECTORDB_FILE_NM']
indexFile = "vectorDB/" + str(vectorDBFileName) + '.faiss'
indexConfig = "vectorDB/" + str(vectorDBFileName) + ".json"
print('File: ', str(indexFile))
print('Config: ', str(indexConfig))
# Also, provide `config_path` parameter if you set it when calling the `save()` method:
new_document_store = FAISSDocumentStore.load(index_path=indexFile, config_path=indexConfig)
# Initialize Retriever
retriever = DensePassageRetriever(document_store=new_document_store,
query_embedding_model=queryModel,
passage_embedding_model=passageModel,
use_gpu=False)
###############################################
### End of Global Section ###
###############################################
class clsRAGOpenAI:
def __init__(self):
self.basePath = cf.conf['DATA_PATH']
self.fileName = cf.conf['FILE_NAME']
self.Ind = cf.conf['DEBUG_IND']
self.subdir = str(cf.conf['OUT_DIR'])
self.base_url = cf.conf['BASE_URL']
self.outputPath = cf.conf['OUTPUT_PATH']
self.vectorDBPath = cf.conf['VECTORDB_PATH']
self.openAIKey = cf.conf['OPEN_AI_KEY']
self.temp = cf.conf['TEMP_VAL']
self.modelName = cf.conf['MODEL_NAME']
self.maxToken = cf.conf['MAX_TOKEN']
def extractHash(self, text):
try:
# Regular expression pattern to match 'Ref: {' followed by a number and then '}'
pattern = r"Ref: \{'(\d+)'\}"
match = re.search(pattern, text)
if match:
return match.group(1)
else:
return None
except Exception as e:
x = str(e)
print('Error: ', x)
return None
def removeSentencesWithNaN(self, text):
try:
# Split text into sentences using regular expression
sentences = re.split('(?<!\w\.\w.)(?<![A-Z][a-z]\.)(?<=\.|\?)\s', text)
# Filter out sentences containing 'nan'
filteredSentences = [sentence for sentence in sentences if 'nan' not in sentence]
# Rejoin the sentences
return ' '.join(filteredSentences)
except Exception as e:
x = str(e)
print('Error: ', x)
return ''
def retrieveDocumentsReader(self, question, top_k=9):
return retriever.retrieve(question, top_k=top_k)
def generateAnswerWithGPT3(self, retrieved_docs, question):
try:
openai.api_key = self.openAIKey
temp = self.temp
modelName = self.modelName
maxToken = self.maxToken
documentsText = " ".join([doc.content for doc in retrieved_docs])
filteredDocs = self.removeSentencesWithNaN(documentsText)
hashValue = self.extractHash(filteredDocs)
print('RAG Docs:: ')
print(filteredDocs)
#prompt = f"Given the following documents: {documentsText}, answer the question accurately based on the above data with the supplied http urls: {question}"
# Set up a chat-style prompt with your data
messages = [
{"role": "system", "content": "You are a helpful assistant, answer the question accurately based on the above data with the supplied http urls. Only relevant content needs to publish. Please do not provide the facts or the texts that results crossing the max_token limits."},
{"role": "user", "content": filteredDocs}
]
# Chat style invoking the latest model
response = openai.ChatCompletion.create(
model=modelName,
messages=messages,
temperature = temp,
max_tokens=maxToken
)
return hashValue, response.choices[0].message['content'].strip().replace('\n','\\n')
except Exception as e:
x = str(e)
print('failed to get from OpenAI: ', x)
return 'Not Available!'
def ragAnswerWithHaystackAndGPT3(self, question):
retrievedDocs = self.retrieveDocumentsReader(question)
return self.generateAnswerWithGPT3(retrievedDocs, question)
def getData(self, strVal):
try:
print('*'*120)
print('Index Your Data for Retrieval:')
print('*'*120)
print('Response from New Docs: ')
print()
hashValue, answer = self.ragAnswerWithHaystackAndGPT3(strVal)
print('GPT3 Answer::')
print(answer)
print('Hash Value:')
print(str(hashValue))
print('*'*240)
print('End Of Use RAG to Generate Answers:')
print('*'*240)
return hashValue, answer
except Exception as e:
x = str(e)
print('Error: ', x)
answer = x
hashValue = 1
return hashValue, answer
Let us understand some of the important block –
Function – ragAnswerWithHaystackAndGPT3():
The ragAnswerWithHaystackAndGPT3 function retrieves relevant documents for a given question using the retrieveDocumentsReader method. It then generates an answer for the query using GPT-3 with the retrieved documents via the generateAnswerWithGPT3 method. The final response is returned.
Function – generateAnswerWithGPT3():
The generateAnswerWithGPT3 function, given a list of retrieved documents and a question, communicates with OpenAI’s GPT-3 to generate an answer. It first processes the documents, filtering and extracting a hash value. Using a chat-style format, it prompts GPT-3 with the processed documents and captures its response. If an error occurs, an error message is printed, and “Not Available!” is returned.
Function – retrieveDocumentsReader():
The retrieveDocumentsReader function takes in a question and an optional parameter, top_k (defaulted to 9). It is called the retriever.retrieve method with the given parameters. The result of the retrieval will generate at max nine responses from the RAG engine, which will be fed to OpenAI.
React:
// App.js
import React, { useState } from 'react';
import axios from 'axios';
import './App.css';
const App = () => {
const [isLoggedIn, setIsLoggedIn] = useState(false);
const [username, setUsername] = useState('');
const [password, setPassword] = useState('');
const [message, setMessage] = useState('');
const [chatLog, setChatLog] = useState([{ sender: 'MuBot', message: 'Welcome to MuBot! Please explore the world of History from our brilliant collections! Do you want to proceed to see the catalog?'}]);
const handleLogin = async (e) => {
e.preventDefault();
try {
const response = await axios.post('http://localhost:5000/login', { username, password });
if (response.status === 200) {
setIsLoggedIn(true);
}
} catch (error) {
console.error('Login error:', error);
}
};
const sendMessage = async (username) => {
if (message.trim() === '') return;
// Create a new chat entry
const newChatEntry = {
sender: 'user',
message: message.trim(),
};
// Clear the input field
setMessage('');
try {
// Make API request to Python-based API
const response = await axios.post('http://localhost:5000/chat', { message: newChatEntry.message }); // Replace with your API endpoint URL
const responseData = response.data;
// Print the response to the console for debugging
console.log('API Response:', responseData);
// Parse the nested JSON from the 'message' attribute
const jsonData = JSON.parse(responseData.message);
// Check if the data contains 'departments'
if (jsonData.departments) {
// Extract the 'departments' attribute from the parsed data
const departments = jsonData.departments;
// Extract the department names and create a single string with line breaks
const botResponseText = departments.reduce((acc, department) => {return acc + department.departmentId + ' ' + department.displayName + '\n';}, '');
// Update the chat log with the bot's response
setChatLog((prevChatLog) => [...prevChatLog, { sender: 'user', message: message }, { sender: 'bot', message: botResponseText },]);
}
else if (jsonData.records)
{
// Data structure 2: Artwork information
const records = jsonData.records;
// Prepare chat entries
const chatEntries = [];
// Iterate through records and extract text, image, and wiki information
records.forEach((record) => {
const textInfo = Object.entries(record).map(([key, value]) => {
if (key !== 'Image' && key !== 'Wiki') {
return `${key}: ${value}`;
}
return null;
}).filter((info) => info !== null).join('\n');
const imageLink = record.Image;
//const wikiLinks = JSON.parse(record.Wiki.replace(/'/g, '"'));
//const wikiLinks = record.Wiki;
const wikiLinks = record.Wiki.split(',').map(link => link.trim());
console.log('Wiki:', wikiLinks);
// Check if there is a valid image link
const hasValidImage = imageLink && imageLink !== '[]';
const imageElement = hasValidImage ? (
<img src={imageLink} alt="Artwork" style={{ maxWidth: '100%' }} />
) : null;
// Create JSX elements for rendering the wiki links (if available)
const wikiElements = wikiLinks.map((link, index) => (
<div key={index}>
<a href={link} target="_blank" rel="noopener noreferrer">
Wiki Link {index + 1}
</a>
</div>
));
if (textInfo) {
chatEntries.push({ sender: 'bot', message: textInfo });
}
if (imageElement) {
chatEntries.push({ sender: 'bot', message: imageElement });
}
if (wikiElements.length > 0) {
chatEntries.push({ sender: 'bot', message: wikiElements });
}
});
// Update the chat log with the bot's response
setChatLog((prevChatLog) => [...prevChatLog, { sender: 'user', message }, ...chatEntries, ]);
}
} catch (error) {
console.error('Error sending message:', error);
}
};
if (!isLoggedIn) {
return (
<div className="login-container">
<h2>Welcome to the MuBot</h2>
<form onSubmit={handleLogin} className="login-form">
<input
type="text"
placeholder="Enter your name"
value={username}
onChange={(e) => setUsername(e.target.value)}
required
/>
<input
type="password"
placeholder="Enter your password"
value={password}
onChange={(e) => setPassword(e.target.value)}
required
/>
<button type="submit">Login</button>
</form>
</div>
);
}
return (
<div className="chat-container">
<div className="chat-header">
<h2>Hello, {username}</h2>
<h3>Chat with MuBot</h3>
</div>
<div className="chat-log">
{chatLog.map((chatEntry, index) => (
<div
key={index}
className={`chat-entry ${chatEntry.sender === 'user' ? 'user' : 'bot'}`}
>
<span className="user-name">{chatEntry.sender === 'user' ? username : 'MuBot'}</span>
<p className="chat-message">{chatEntry.message}</p>
</div>
))}
</div>
<div className="chat-input">
<input
type="text"
placeholder="Type your message..."
value={message}
onChange={(e) => setMessage(e.target.value)}
onKeyPress={(e) => {
if (e.key === 'Enter') {
sendMessage();
}
}}
/>
<button onClick={sendMessage}>Send</button>
</div>
</div>
);
};
export default App;
Please find some of the important logic –
Function – handleLogin():
The handleLogin asynchronous function responds to an event by preventing its default action. It attempts to post a login request with a username and password to a local server endpoint. If the response is successful with a status of 200, it updates a state variable to indicate a successful login; otherwise, it logs any encountered errors.
Function – sendMessage():
The sendMessage asynchronous function is designed to handle the user’s chat interaction:
This function enables interactive chat with bot responses that vary based on the nature of the data received from the API.
DIRECTORY STRUCTURES:
Let us explore the directory structure starting from the parent to some of the important child folder should look like this –
So, finally, we’ve done it.
I know that this post is relatively bigger than my earlier post. But, I think, you can get all the details once you go through it.
You will get the complete codebase in the following GitHub link.
I’ll bring some more exciting topics in the coming days from the Python verse. Please share & subscribe to my post & let me know your feedback.
Till then, Happy Avenging! 🙂
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. Some of the images (except my photo) we’ve used are available over the net. We don’t claim ownership of these images. There is always room for improvement & especially in the prediction quality.
Today, I’m very excited to demonstrate an effortless & new way to fine-tune the GPT-3 model using Python with the help of my new build (unpublished) PyPi package. In this post, I plan to deal with the custom website link as a response from this website depending upon the user queries with the help of the OpenAI-based tuned model.
In this post, I’ve directly subscribed to OpenAI & I’m not using OpenAI from Azure. However, I’ll explore that in the future as well.
Before I explain the process to invoke this new library, why not view the demo first & then discuss it?
Isn’t it exciting? Finally, we can efficiently handle your custom website URL using OpenAI tuned model.
What is ChatGPT?
ChatGPT is an advanced artificial intelligence language model developed by OpenAI based on the GPT-4 architecture. As an AI model, it is designed to understand and generate human-like text-based on the input it receives. ChatGPT can engage in various tasks, such as answering questions, providing recommendations, creating content, and simulating conversation. While it is highly advanced and versatile, it’s important to note that ChatGPT’s knowledge is limited to the data it was trained on, with a cutoff date of September 2021.
When to tune GPT model?
Tuning a GPT or any AI model might be necessary for various reasons. Here are some common scenarios when you should consider adjusting or fine-tuning a GPT model:
Remember that tuning or fine-tuning a GPT model requires access to appropriate data and computational resources and an understanding of the model’s architecture and training techniques. Additionally, monitoring and evaluating the model’s performance after fine-tuning is essential to ensure that the desired improvements have been achieved.
FLOW OF EVENTS:
Let us look at the flow diagram as it captures the sequence of events that unfold as part of the process.
The initial Python-based client interacts with the tuned OpenAI models. This process enables it to get a precise response with custom data in a very convenient way. So that anyone can understand.
SOURCE DATA:
Let us understand how to feed the source data as it will deal with your website URL link.
The first data that we are going to talk about is the one that contains the hyperlink. Let us explore the sample here.
From the above diagram, one can easily understand that the application will interpret a unique hash number associated with a specific URL. This data will be used to look up the URL after the OpenAI response from the tuned model as a result of any user query.
Now, let us understand the actual source data.
If we closely check, we’ll see the source file contains two columns – prompt & completion. And the website reference is put inside the curly braces as shown – “{Hash Code that represents your URL}.”
During the response, the newly created library replaces the hash value with the correct URL after the successful lookup & presents the complete answer.
CODE:
Why don’t we go through the code made accessible due to this new library for this particular use case?
################################################ | |
#### Written By: SATYAKI DE #### | |
#### Written On: 15-May-2020 #### | |
#### Modified On: 21-Feb-2023 #### | |
#### #### | |
#### Objective: This script is a config #### | |
#### file, contains all the keys for #### | |
#### OpenAI fine-tune projects. #### | |
#### #### | |
################################################ | |
import os | |
import platform as pl | |
class clsConfigClient(object): | |
Curr_Path = os.path.dirname(os.path.realpath(__file__)) | |
os_det = pl.system() | |
if os_det == "Windows": | |
sep = '\\' | |
else: | |
sep = '/' | |
conf = { | |
'APP_ID': 1, | |
'ARCH_DIR': Curr_Path + sep + 'arch' + sep, | |
'PROFILE_PATH': Curr_Path + sep + 'profile' + sep, | |
'LOG_PATH': Curr_Path + sep + 'log' + sep, | |
'DATA_PATH': Curr_Path + sep + 'data' + sep, | |
'TEMP_PATH': Curr_Path + sep + 'temp' + sep, | |
'MODEL_DIR': 'model', | |
'APP_DESC_1': 'ChatGPT Training!', | |
'DEBUG_IND': 'N', | |
'INIT_PATH': Curr_Path, | |
'FILE_NAME': '2023-4-14-WP.csv', | |
'LKP_FILE_NAME': 'HyperDetails.csv', | |
'TEMP_FILE_NAME': 'chatGPTData.jsonl', | |
'TITLE': "GPT-3 Training!", | |
'PATH' : Curr_Path, | |
'OUT_DIR': 'data', | |
'OPEN_API_KEY': 'sk-hdhrujfrkfjfjfjfhjfjfisososT&jsdgL6KIxx', | |
'MODEL_CD':'davinci', | |
'URL': 'https://api.openai.com/v1/fine-tunes/', | |
'EPOCH': 10, | |
'SUFFIX': 'py-saty', | |
'EXIT_KEYWORD': 'bye' | |
} |
Some of the important entries that will require later are as follows –
'FILE_NAME': '2023-4-14-WP.csv',
'LKP_FILE_NAME': 'HyperDetails.csv',
'OPEN_API_KEY': 'sk-hdhrujfrkfjfjfjfhjfjfisososT&jsdgL6KIxx',
'MODEL_CD':'davinci',
'URL': 'https://api.openai.com/v1/fine-tunes/',
'EXIT_KEYWORD': 'bye'
We’ll discuss these entries later.
##################################################### | |
#### Written By: SATYAKI DE #### | |
#### Written On: 12-Feb-2023 #### | |
#### Modified On 16-Feb-2023 #### | |
#### #### | |
#### Objective: This is the main calling #### | |
#### python script that will invoke the #### | |
#### newly created fine-tune GPT-3 enabler. #### | |
#### #### | |
##################################################### | |
import pandas as p | |
import clsL as cl | |
from clsConfigClient import clsConfigClient as cf | |
import datetime | |
import clsTrainModel3 as tm | |
# Disbling Warning | |
def warn(*args, **kwargs): | |
pass | |
import warnings | |
warnings.warn = warn | |
###################################### | |
### Get your global values #### | |
###################################### | |
debug_ind = 'Y' | |
#tModel = tm.clsTrainModel() | |
tModel = tm.clsTrainModel3() | |
# Initiating Logging Instances | |
clog = cl.clsL() | |
data_path = cf.conf['DATA_PATH'] | |
data_file_name = cf.conf['FILE_NAME'] | |
###################################### | |
#### Global Flag ######## | |
###################################### | |
###################################### | |
### Wrapper functions to invoke ### | |
### the desired class from newly ### | |
### built class. ### | |
###################################### | |
###################################### | |
### End of wrapper functions. ### | |
###################################### | |
def main(): | |
try: | |
var = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S") | |
print('*'*120) | |
print('Start Time: ' + str(var)) | |
print('*'*120) | |
FullFileName = data_path + data_file_name | |
r1 = tModel.trainModel(FullFileName) | |
if r1 == 0: | |
print('Successfully Trained!') | |
else: | |
print('Failed to Train!') | |
#clog.logr(OutPutFileName, debug_ind, df, subdir) | |
print('*'*120) | |
var1 = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S") | |
print('End Time: ' + str(var1)) | |
except Exception as e: | |
x = str(e) | |
print('Error: ', x) | |
if __name__ == "__main__": | |
main() |
Following are the key snippet from the above script –
data_path = cf.conf['DATA_PATH']
data_file_name = cf.conf['FILE_NAME']
And, then –
tModel = tm.clsTrainModel3()
FullFileName = data_path + data_file_name
r1 = tModel.trainModel(FullFileName)
As one can see, the package needs only the source data file to fine-tune GPT-3 model.
##################################################### | |
#### Written By: SATYAKI DE #### | |
#### Written On: 12-Feb-2023 #### | |
#### Modified On 16-Feb-2023 #### | |
#### #### | |
#### Objective: This is the main calling #### | |
#### python script that will invoke the #### | |
#### newly created fine-tune job status inside #### | |
#### the OpenAI environment. #### | |
##################################################### | |
import clsL as cl | |
from clsConfigClient import clsConfigClient as cf | |
import datetime | |
import clsTestModel3 as tm | |
# Disbling Warning | |
def warn(*args, **kwargs): | |
pass | |
import warnings | |
warnings.warn = warn | |
###################################### | |
### Get your global values #### | |
###################################### | |
debug_ind = 'Y' | |
# Initiating Logging Instances | |
clog = cl.clsL() | |
tmodel = tm.clsTestModel3() | |
url_part = cf.conf['URL'] | |
open_api_key = cf.conf['OPEN_API_KEY'] | |
###################################### | |
#### Global Flag ######## | |
###################################### | |
def main(): | |
try: | |
var = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S") | |
print('*'*120) | |
print('Start Time: ' + str(var)) | |
print('*'*120) | |
# Example usage | |
input_text = str(input("Please provide the fine tune Id (Start with ft-*): ")) | |
url = url_part + input_text | |
print('URL: ', url) | |
r1 = tmodel.checkStat(url, open_api_key) | |
if r1 == 0: | |
print('Successfully checked the status of tuned GPT-3 model.') | |
else: | |
print('Failed to check the status of the tuned GPT-3 model.') | |
print('*'*120) | |
var1 = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S") | |
print('End Time: ' + str(var1)) | |
except Exception as e: | |
x = str(e) | |
print('Error: ', x) | |
if __name__ == "__main__": | |
main() |
To check the status of the fine-tuned job inside the OpenAI environment, one needs to provide the fine tune id, which generally starts with -> “ft-*.” One would get this value after the train script’s successful run.
Some of the other key snippets are –
tmodel = tm.clsTestModel3()
url_part = cf.conf['URL']
open_api_key = cf.conf['OPEN_API_KEY']
And, then –
input_text = str(input("Please provide the fine tune Id (Start with ft-*): "))
url = url_part + input_text
print('URL: ', url)
r1 = tmodel.checkStat(url, open_api_key)
The above snippet is self-explanatory as one is passing the fine tune id along with the OpenAI API key.
##################################################### | |
#### Written By: SATYAKI DE #### | |
#### Written On: 12-Feb-2023 #### | |
#### Modified On 19-Apr-2023 #### | |
#### #### | |
#### Objective: This is the main calling #### | |
#### python script that will invoke the #### | |
#### newly created class that will test the #### | |
#### tuned model output. #### | |
##################################################### | |
import clsL as cl | |
from clsConfigClient import clsConfigClient as cf | |
import datetime | |
import pandas as p | |
import clsTestModel3 as tm | |
# Disbling Warning | |
def warn(*args, **kwargs): | |
pass | |
import warnings | |
warnings.warn = warn | |
###################################### | |
### Get your global values #### | |
###################################### | |
debug_ind = 'Y' | |
# Initiating Logging Instances | |
clog = cl.clsL() | |
tmodel = tm.clsTestModel3() | |
open_api_key = cf.conf['OPEN_API_KEY'] | |
lkpDataPath = cf.conf['DATA_PATH'] | |
lkpFileName = cf.conf['LKP_FILE_NAME'] | |
###################################### | |
#### Global Flag ######## | |
###################################### | |
def main(): | |
try: | |
var = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S") | |
print('*' * 120) | |
print('Start Time: ' + str(var)) | |
print('*' * 120) | |
LookUpFileName = lkpDataPath + lkpFileName | |
r1 = tmodel.testModel(LookUpFileName, open_api_key) | |
if r1 == 0: | |
print('Successfully tested the tuned GPT-3 model.') | |
else: | |
print('Failed to test the tuned GPT-3 model.') | |
var1 = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S") | |
print('End Time: ' + str(var1)) | |
except Exception as e: | |
x = str(e) | |
print('Error: ', x) | |
if __name__ == "__main__": | |
main() |
Some of the key entries from the above snippet are as follows –
tmodel = tm.clsTestModel3()
open_api_key = cf.conf['OPEN_API_KEY']
lkpDataPath = cf.conf['DATA_PATH']
lkpFileName = cf.conf['LKP_FILE_NAME']
And, then –
LookUpFileName = lkpDataPath + lkpFileName
r1 = tmodel.testModel(LookUpFileName, open_api_key)
In the above lines, the application gets the correct URL value from the look file we’ve prepared for this specific use case.
##################################################### | |
#### Written By: SATYAKI DE #### | |
#### Written On: 12-Feb-2023 #### | |
#### Modified On 21-Feb-2023 #### | |
#### #### | |
#### Objective: This is the main calling #### | |
#### python script that will invoke the #### | |
#### newly created delete model methods for #### | |
#### OpenAI. #### | |
##################################################### | |
import clsL as cl | |
from clsConfigClient import clsConfigClient as cf | |
import datetime | |
import clsTestModel3 as tm | |
# Disbling Warning | |
def warn(*args, **kwargs): | |
pass | |
import warnings | |
warnings.warn = warn | |
###################################### | |
### Get your global values #### | |
###################################### | |
debug_ind = 'Y' | |
# Initiating Logging Instances | |
clog = cl.clsL() | |
tmodel = tm.clsTestModel3() | |
open_api_key = cf.conf['OPEN_API_KEY'] | |
###################################### | |
#### Global Flag ######## | |
###################################### | |
def main(): | |
try: | |
var = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S") | |
print('*' * 120) | |
print('Start Time: ' + str(var)) | |
print('*' * 120) | |
r1 = tmodel.delOldModel(open_api_key) | |
if r1 == 0: | |
print('Successfully checked the status of tuned GPT-3 model.') | |
else: | |
print('Failed to check the status of the tuned GPT-3 model.') | |
var1 = datetime.datetime.now().strftime("%Y-%m-%d_%H-%M-%S") | |
print('End Time: ' + str(var1)) | |
except Exception as e: | |
x = str(e) | |
print('Error: ', x) | |
if __name__ == "__main__": | |
main() |
Some of the key snippets from the above scripts are –
tmodel = tm.clsTestModel3()
open_api_key = cf.conf['OPEN_API_KEY']
And, then –
r1 = tmodel.delOldModel(open_api_key)
We’ve demonstrated that using a straightforward method, one can delete any old tuned model from OpenAI that is no longer required.
KEY FEATURES TO CONSIDER DURING TUNING:
COST FACTOR:
Before we discuss the actual spending, let us understand the tested data volume to train & tune the model.
So, we’re talking about a total size of 500 KB (at max). And, we did 10 epochs during the training as you can see from the config file mentioned above.
So, it is pretty expensive. Use it wisely.
So, finally, we’ve done it.
I know that this post is relatively bigger than my earlier post. But, I think, you can get all the details once you go through it.
You will get the complete codebase in the following GitHub link.
I’ll bring some more exciting topics in the coming days from the Python verse. Please share & subscribe to my post & let me know your feedback.
Till then, Happy Avenging! 🙂
Note: All the data & scenarios posted here are representational data & scenarios & available over the internet & for educational purposes only. Some of the images (except my photo) we’ve used are available over the net. We don’t claim ownership of these images. There is always room for improvement & especially in the prediction quality.
You must be logged in to post a comment.