Managing mesh-APIs a few best practices including circuit-breaker & more.

Hi Guys!

Today, We will be discussing some of the better use-cases where we want to implement better solutions, which helps our entire Microservice/API integration painless.

As we drive towards more & more real-time applications such as mobile apps, IoT, and bots, we are engaging more microservice/API components in our technology stack. Moreover, We have witnessed that on many occasions, a full grown microservice environment leads to many unforeseen scenarios, which was not predicted or captured correctly.

To understand how a mesh-API architecture looks like, let us take one sample example for our case. Furthermore, what kind of problem we might face & how we can address.

Let us consider Airline online check-in as an example here.

There are following components are taking place irrespective of any airline ->

  • Filling-up primary information such as First Name, Last Name, Address, Contact details, Passport & Booking No.
  • We will have to choose a seat.
  • We will need to provide the input for a meal (However, this is optional as the domestic flight may or may not have this based on the distance & budget)
  • Finally, generations of Boarding-Pass.

If we decoupled our application here, we could create four inter-connected microservices to cater to our requirements. And this is shown in the given diagram –

Normal Microservice operations

From the above diagram, we can see that customer invokes a couple of related but independent services to finish the process of generating a Boarding-Pass. As per the diagram, initial requests reach the main Onboarding API & that results in cascading service calls & their successful response. In a developer’s term, this one we call it a “Happy Scenario!”

Nevertheless, developers do not always get this kind of rosy scenario. Sometimes, we encountered a scenario like this –

Cascading failure in microservice framework

Suppose we see an issue with our Boarding-pass generation service, which may encounter some network or any fatal issue, which leads to a chain of failure to its parent services. Moreover, the result will be disastrous.

How can we handle such scenarios? There are a few guidelines that we can explore.

  1. Time-Out trigger while calling any service/s
  2. Building Asynchronous service
  3. Implementation of Circuit-Breaker

We can also use the standard workflow management or custom build workflow manager to improve that further (We will discuss this later in this blog).

We understood that our parent service should not wait for a longer time, where there is a significant delay from the child service. Otherwise, it will have a cascading effect & bring down the entire array of services and it.

However, the use of Circuit-Breaker is an advanced & better way to handle such scenarios. To keep this post small, We are not going to share all the codebase here. We will examine mainly two base codes.

Please find the current directory structure –

Current directory structure

Let us consider implementing without the Circuit-Breaker. We have created the following functions & a custom Python library deployed for this demo. However, We will not discuss anything in this library today. We will release it in PyPi & share the post the update & detail functionality here later. This library is a Python-based lightweight workflow manager, which will continuously monitor any failed service & when connected – it will trigger it when connected.

Use of Python-based Workflow Manager

From the above diagram, we can see that if we force “time-out” while calling our services, we can use this lightweight framework later whenever that service wake-up or establish a connection.

Please find the WhatsApp delivery through our microservice framework using this lightweight workflow framework.

Sending WhatsApp message using Python-based Workflow Manager

We have used Visual studio code editor for this entire development of azure functions. And, the screen should look something like this –

Development Editor

We’ll discuss one the following two services –

  1. getSeatSelection
  2. getBoardingPass

We will also discuss the first script twice – one is with the standard time-out approach & another with the circuit-breaker.

  1. getSeatSelection ( This Microservice receives the inputs from its parent microservice & confirm the preferred seat-no selected by the passengers. )
##############################################
#### Written By: SATYAKI DE               ####
#### Written On: 05-Oct-2020              ####
#### Modified On 05-Oct-2020              ####
####                                      ####
#### Objective: Scripts to choose the     ####
#### seat for the passenger.              ####
##############################################

import logging
import json
import time
import os
import ssl
import requests

import azure.functions as func

def callService(inputJson, urlPost, retryNo, eventFlag, maxRetryNo, maxTimeOut):
    # Invoking getBoardingPass API
    try:
        # Bypassing SSL Authentication
        try:
            _create_unverified_https_context = ssl._create_unverified_context
        except AttributeError:
            # Legacy python that doesn't verify HTTPS certificates by default
            pass
        else:
            # Handle target enviornment that doesn't support HTTPS verification
            ssl._create_default_https_context = _create_unverified_https_context

        # Providing the url bearer token
        url = urlPost

        json_data = inputJson
        headers = {'content-type': "application/json", 'cache-control': "no-cache"}

        # Failed case retry
        retries = retryNo
        success = False

        return_description = 'N/A'
        statusVal = 'Failed'

        try:
            while not success:
                try:
                    # Getting response from web service
                    try:
                        strCheckMsg = 'Sending Payload - ' + str(retries)
                        logging.info(strCheckMsg)
                        
                        response = requests.request("POST", url, data=json_data, headers=headers, verify=False, timeout=maxTimeOut)

                        resp_dict = json.loads(response.text)

                        statusVal = resp_dict['status']
                        return_description = resp_dict['description']

                        logging.info(statusVal)

                    except Exception as e:
                        x = str(e)
                        logging.info(x)
                        success = False

                    if (eventFlag == 'Boarding-Pass'):
                        str_R = "Boarding-Pass Json Response:: " + str(response.text)
                    else:
                        str_R = "Invalid flag options "
                    
                    logging.info((str_R))

                    if len(response.text) > 80:
                        if str(response.status_code)[:1] == '2':
                            success = True
                        else:
                            wait = retries * 2
                            str_R1 = "retries Fail! Waiting " + str(wait) + " seconds and retrying!"
                            logging.info(str_R1)
                            time.sleep(wait)
                            retries += 1

                        # Checking maximum retries
                        if retries >= maxRetryNo:
                            success = True
                            raise  Exception
                    else:
                        if str(response.status_code)[:1] == '2':
                            success = True
                        else:
                            # Checking maximum entries
                            if retries >= maxRetryNo:
                                success = True
                                raise  Exception

                            retries += 1
                except:
                    strVal = 'Retrying - ' + str(retries)
                    logging.info(strVal)
                    # Checking maximum entries
                    if retries >= maxRetryNo:
                        success = True
                        raise  Exception

                    retries += 1

            selection_flag = 'Y'

            # Forming return JSON
            jsonRet = {
                "eventFlag": eventFlag,
                "status":statusVal,
                "description":return_description,
                "computeFlag":selection_flag
            }

            xval = json.dumps(jsonRet)

            return xval

        except (ConnectionError, TimeoutError, InterruptedError) as e:
            str_R8 = "Response from Server: " + str(response)
            logging.info(str_R8)

            str_R9 = "Sending payload to Webservice!"
            logging.info(str_R9)
            selection_flag = 'N'

            # Forming return JSON
            jsonRet = {
                "eventFlag": eventFlag,
                "status":"Lost",
                "description":"Timed-Out or Connection Error or any I/O issue preventing the process. We're working on this!",
                "computeFlag":selection_flag
            }

            xval = json.dumps(jsonRet)

            return xval

        except ValueError as e:
            x = str(e)
            logging.info(x)

            selection_flag = 'N'

            # Forming return JSON
            jsonRet = {
                "eventFlag": eventFlag,
                "status":"Failed",
                "description":"Please check the internal parameters compatibility with it's data-type!",
                "computeFlag":selection_flag
            }

            xval = json.dumps(jsonRet)

            return xval

        except Exception as e:
            x = str(e)
            logging.info(x)

            selection_flag = 'N'

            # Forming return JSON
            jsonRet = {
                "eventFlag": eventFlag,
                "status":"Accepted",
                "description":"Maximum rerties to the Boarding-Pass service is now reached. We're working offline to get that for you. Please rerty after 4 hours in case if you haven't received it or you can directly call customer care number!",
                "computeFlag":selection_flag
            }

            xval = json.dumps(jsonRet)

            return xval


    except Exception as e:
        x = str(e)
        logging.info(x)

        selection_flag = 'N'

        # Forming return JSON
        jsonRet = {
            "eventFlag": eventFlag,
            "status":"Failed",
            "description":"DevOps Engineer is looking into this. Please try after some time!",
            "computeFlag":selection_flag
        }

        xval = json.dumps(jsonRet)

        return xval

def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Invoking Seat-Selection generation service.')

    # Getting System Variable from local settings file
    strVal = 'Parameter Recieved :' + os.environ['maxRetry']
    
    logging.info(strVal)

    # Capturing parameters from local settings
    max_retries = int(os.environ['maxRetry'])
    c_url = os.environ['cUrlBP']

    # Printing input Payload
    str_val = 'Input Payload:: ' + str(req.get_json())
    logging.info(str_val)

    strMainCheck = str(req.get_json())

    # variable
    x_status = 'Success'

    if (strMainCheck != ''):

        # Capturing individual elements

        sourceLeg = req.params.get('sourceLeg')
        destinationLeg = req.params.get('destinationLeg')
        boardingClass = req.params.get('boardingClass')
        preferredSeatNo = req.params.get('preferredSeatNo')
        travelPassport = req.params.get('travelPassport')
        bookingNo = req.params.get('bookingNo')
        travelerEmail = req.params.get('travelerEmail')
        travelerMobile = req.params.get('travelerMobile')

        # Checking Individual Elements

        if not sourceLeg:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                sourceLeg = req_body.get('sourceLeg')

        if not destinationLeg:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                destinationLeg = req_body.get('destinationLeg')

        if not boardingClass:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                boardingClass = req_body.get('boardingClass')

        if not preferredSeatNo:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                preferredSeatNo = req_body.get('preferredSeatNo')

        if not travelPassport:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                travelPassport = req_body.get('travelPassport')
        
        if not bookingNo:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                bookingNo = req_body.get('bookingNo')

        if not travelerEmail:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                travelerEmail = req_body.get('travelerEmail')

        if not travelerMobile:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                travelerMobile = req_body.get('travelerMobile')


        if (
            (sourceLeg != '') & (destinationLeg != '') & (boardingClass != '') & 
            (preferredSeatNo != '') & (travelPassport != '') & (bookingNo != '') &
            ((travelerEmail != '') | (travelerMobile != ''))
            ):

            x_description = "Seat-Selection Successfully Processed!"

            # Preparing Payload for boarding-pass
            json_str_boarding = {
                "sourceLeg": sourceLeg,
                "destinationLeg": destinationLeg,
                "boardingClass": boardingClass,
                "confirmedSeatNo": preferredSeatNo,
                "travelPassport": travelPassport,
                "bookingNo": bookingNo,
                "travelerEmail": travelerEmail,
                "travelerMobile": travelerMobile
                        }

            jsonBoardingInput = json.dumps(json_str_boarding)

            eventFlag = 'Boarding-Pass'
            retryNoBase = 1
            maxTimeOut = 2.5

            # Invoking getBoardingPass API
            microCallsBP = callService(jsonBoardingInput, c_url, retryNoBase, eventFlag, max_retries, maxTimeOut)
            
            # Extracting seat_selection_flag
            resp_dict = json.loads(microCallsBP)
            x_status = resp_dict['status']
            x_description = resp_dict['description']

            # Capturing description value based on the logic
            logging.info(x_description)

            # Formatting return Payload
            json_str = {
                "description": x_description,
                "status": x_status,
                "sourceLeg": sourceLeg,
                "destinationLeg": destinationLeg,
                "boardingClass": boardingClass,
                "confirmedSeatNo": preferredSeatNo,
                "travelPassport": travelPassport,
                "bookingNo": bookingNo,
                "travelerEmail": travelerEmail,
                "travelerMobile": travelerMobile
                        }

            xval = json.dumps(json_str)

            return func.HttpResponse(xval, status_code=200)
        else:
            json_str = {
                "description": "Missing mandatory Email or Phone Number!",
                "status": "Failed",
                "sourceLeg": sourceLeg,
                "destinationLeg": destinationLeg,
                "boardingClass": boardingClass,
                "confirmedSeatNo": preferredSeatNo,
                "travelPassport": travelPassport,
                "bookingNo": bookingNo,
                "travelerEmail": travelerEmail,
                "travelerMobile": travelerMobile
            }

            xval = json.dumps(json_str)
            return func.HttpResponse(xval,status_code=200)
    else:
        json_str = {
            "description": "Missing entire payload!",
            "status": "Failed",
            "sourceLeg": "N/A",
            "destinationLeg": "N/A",
            "boardingClass": "N/A",
            "confirmedSeatNo": "N/A",
            "travelPassport": "N/A",
            "bookingNo": "N/A",
            "travelerEmail": "N/A",
            "travelerMobile": "N/A"
        }

        xval = json.dumps(json_str)
        return func.HttpResponse(xval,status_code=200)

There are a few key snippets that we would like to discuss here.

# Getting response from web service
try:
    strCheckMsg = 'Sending Payload - ' + str(retries)
    logging.info(strCheckMsg)
    
    response = requests.request("POST", url, data=json_data, headers=headers, verify=False, timeout=maxTimeOut)

    resp_dict = json.loads(response.text)

    statusVal = resp_dict['status']
    return_description = resp_dict['description']

    logging.info(statusVal)

except Exception as e:
    x = str(e)
    logging.info(x)
    success = False

From the above snippet, we can see that we are using time-out based on our understanding of response time or the max SLA agreed upon for that particular service.

If we get the response, we are capturing the status as well as the description from its child service & capturing it in our log, which will look something like this –

Azure Monitor of Seat-Selection service

2. getBoardingPass ( This Microservice receives the input from its parent service & confirm the autogenerated boarding pass shared with their WhatsApp number.)

##############################################
#### Written By: SATYAKI DE               ####
#### Written On: 02-Oct-2020              ####
#### Modified On 04-Oct-2020              ####
####                                      ####
#### Objective: Scripts to generate the   ####
#### boarding pass for the passenger.     ####
##############################################

import logging
import json
import hashlib
import time
import os
import random

from twilio.rest import Client
from twilio.base.exceptions import TwilioRestException

import azure.functions as func


def main(req: func.HttpRequest) -> func.HttpResponse:
    try:
        logging.info('Invoking Boarding-Pass generation service.')

        # Getting System Variable from local settings file
        strVal = 'Parameter Recieved :' + os.environ['timeOut']
        
        logging.info(strVal)
        sleep_time = int(os.environ['timeOut'])

        # Printing input Payload
        str_val = 'Input Payload:: ' + str(req.get_json())
        logging.info(str_val)

        strMainCheck = str(req.get_json())

        if (strMainCheck != ''):

            # Capturing individual elements

            sourceLeg = req.params.get('sourceLeg')
            destinationLeg = req.params.get('destinationLeg')
            boardingClass = req.params.get('boardingClass')
            confirmedSeatNo = req.params.get('confirmedSeatNo')
            travelPassport = req.params.get('travelPassport')
            bookingNo = req.params.get('bookingNo')
            travelerEmail = req.params.get('travelerEmail')
            travelerMobile = req.params.get('travelerMobile')

            # Checking Individual Elements

            if not sourceLeg:
                try:
                    req_body = req.get_json()
                except ValueError:
                    pass
                else:
                    sourceLeg = req_body.get('sourceLeg')

            if not destinationLeg:
                try:
                    req_body = req.get_json()
                except ValueError:
                    pass
                else:
                    destinationLeg = req_body.get('destinationLeg')

            if not boardingClass:
                try:
                    req_body = req.get_json()
                except ValueError:
                    pass
                else:
                    boardingClass = req_body.get('boardingClass')

            if not confirmedSeatNo:
                try:
                    req_body = req.get_json()
                except ValueError:
                    pass
                else:
                    confirmedSeatNo = req_body.get('confirmedSeatNo')

            if not travelPassport:
                try:
                    req_body = req.get_json()
                except ValueError:
                    pass
                else:
                    travelPassport = req_body.get('travelPassport')
            
            if not bookingNo:
                try:
                    req_body = req.get_json()
                except ValueError:
                    pass
                else:
                    bookingNo = req_body.get('bookingNo')

            if not travelerEmail:
                try:
                    req_body = req.get_json()
                except ValueError:
                    pass
                else:
                    travelerEmail = req_body.get('travelerEmail')

            if not travelerMobile:
                try:
                    req_body = req.get_json()
                except ValueError:
                    pass
                else:
                    travelerMobile = req_body.get('travelerMobile')

            # Mimicking network latency or issues
            random_sleep_time = random.randint(1, sleep_time)
            time.sleep(random_sleep_time)

            child_cond_time = float(os.environ['timeOutChild'])

            # Important Details capture in log
            strRandSleepTime = 'Network Generated delay: ' + str(random_sleep_time) + ' sec'
            logging.info(strRandSleepTime)

            strThreadTimeOut = 'Orphan microservice kill time: ' + str(child_cond_time) + ' sec'
            logging.info(strThreadTimeOut)

            # Handling orphan microservice
            if (random_sleep_time > child_cond_time):
                strCondChk = 'Rasing exception to avoid orphan microservice call!'
                logging.info(strCondChk)
                raise Exception

            if (
                (sourceLeg != '') & (destinationLeg != '') & (boardingClass != '') & 
                (confirmedSeatNo != '') & (travelPassport != '') & (bookingNo != '') &
                ((travelerEmail != '') | (travelerMobile != ''))
                ):

                # Generating simulated barcode
                # Concatenating Key Values
                strImpVal = str(sourceLeg)+str(destinationLeg)+str(boardingClass)+str(confirmedSeatNo)+str(travelPassport)+str(bookingNo)+str(travelerEmail)+str(travelerMobile)

                # Printing Concatenated Value
                logging.info(strImpVal)

                # m.update(strImpVal)
                xbarcode = hashlib.md5(strImpVal.encode("utf").strip()).hexdigest().upper()

                # Formatting return Payload
                json_str = {
                    "description": "Boarding-Pass Successfully Generated!",
                    "status": "Success",
                    "sourceLeg": sourceLeg,
                    "destinationLeg": destinationLeg,
                    "boardingClass": boardingClass,
                    "confirmedSeatNo": confirmedSeatNo,
                    "travelPassport": travelPassport,
                    "bookingNo": bookingNo,
                    "travelerEmail": travelerEmail,
                    "travelerMobile": travelerMobile,
                    "barCode": xbarcode
                            }

                xval = json.dumps(json_str)

                # Calling Twilio API to send WhatsApp Message
                try:
                    strComAPI = 'Calling WhatsApp API'
                    logging.info(strComAPI)

                    account_sid = str(os.environ['accountSid']) 
                    auth_token = os.environ['authToken'] 
                    mobPr = os.environ['mobPredicate']

                    fromMob = mobPr + os.environ['fromMobile'] 
                    toMob = mobPr + travelerMobile

                    client = Client(account_sid, auth_token)

                    msgBody = 'Your Boarding-Pass from ' + sourceLeg + ' to ' + destinationLeg + ' are as follows --> Boarding Class (' + boardingClass + ') - ' + 'Seat (' + confirmedSeatNo + ') - Gate (A' + str(random_sleep_time) + ') - ' + 'Bar Code (' +  xbarcode + ')'

                    message = client.messages.create(from_=fromMob, body=msgBody, to=toMob)

                    msgStat = message.status

                    if (msgStat.upper() == 'QUEUED') | (msgStat.upper() == 'DELIEVERD') | (msgStat.upper() == 'SUCCESS'):
                        return func.HttpResponse(xval, status_code=200)
                    else:
                        # Formatting return Payload
                        json_str = {
                            "description": "Encountered some technical issue. Don't worry - we're working on it!",
                            "status": "Accepted",
                            "sourceLeg": sourceLeg,
                            "destinationLeg": destinationLeg,
                            "boardingClass": boardingClass,
                            "confirmedSeatNo": confirmedSeatNo,
                            "travelPassport": travelPassport,
                            "bookingNo": bookingNo,
                            "travelerEmail": travelerEmail,
                            "travelerMobile": travelerMobile,
                            "barCode": xbarcode
                                    }

                        xval = json.dumps(json_str)

                        return func.HttpResponse(xval, status_code=200)

                except TwilioRestException as e:
                    x = str(e)
                    logging.info(x)

                    # Formatting return Payload
                    json_str = {
                        "description": "Encountered some technical issue. Don't worry - we're working on it!",
                        "status": "Delivery Failed!",
                        "sourceLeg": sourceLeg,
                        "destinationLeg": destinationLeg,
                        "boardingClass": boardingClass,
                        "confirmedSeatNo": confirmedSeatNo,
                        "travelPassport": travelPassport,
                        "bookingNo": bookingNo,
                        "travelerEmail": travelerEmail,
                        "travelerMobile": travelerMobile,
                        "barCode": xbarcode
                                }

                    xval = json.dumps(json_str)

                    return func.HttpResponse(xval, status_code=200)
                
            else:
                xbarcode = 'Not Generated!'
                json_str = {
                    "description": "Missing mandatory Email or Phone Number!",
                    "status": "Failed",
                    "sourceLeg": sourceLeg,
                    "destinationLeg": destinationLeg,
                    "boardingClass": boardingClass,
                    "confirmedSeatNo": confirmedSeatNo,
                    "travelPassport": travelPassport,
                    "bookingNo": bookingNo,
                    "travelerEmail": travelerEmail,
                    "travelerMobile": travelerMobile,
                    "barCode": xbarcode
                }

                xval = json.dumps(json_str)
                return func.HttpResponse(xval,status_code=200)
        else:
            xbarcode = 'Not Generated!'
            json_str = {
                "description": "Missing entire payload!",
                "status": "Failed",
                "sourceLeg": "N/A",
                "destinationLeg": "N/A",
                "boardingClass": "N/A",
                "confirmedSeatNo": "N/A",
                "travelPassport": "N/A",
                "bookingNo": "N/A",
                "travelerEmail": "N/A",
                "travelerMobile": "N/A",
                "barCode": xbarcode
            }

            xval = json.dumps(json_str)
            return func.HttpResponse(xval,status_code=200)
    except Exception as e:
        x = str(e)
        logging.info(x)

        x_description = 'Time-out due to network delay!'
        logging.info(x_description)

        xbarcode = 'Not Generated!'
        json_str = {
            "description": x_description,
            "status": "Failed",
            "sourceLeg": "N/A",
            "destinationLeg": "N/A",
            "boardingClass": "N/A",
            "confirmedSeatNo": "N/A",
            "travelPassport": "N/A",
            "bookingNo": "N/A",
            "travelerEmail": "N/A",
            "travelerMobile": "N/A",
            "barCode": xbarcode
        }

        xval = json.dumps(json_str)
        return func.HttpResponse(xval,status_code=200)



Let us explore some of the vital snippets –

# Mimicking network latency or issues
random_sleep_time = random.randint(1, sleep_time)
time.sleep(random_sleep_time)

child_cond_time = float(os.environ['timeOutChild'])

# Important Details capture in log
strRandSleepTime = 'Network Generated delay: ' + str(random_sleep_time) + ' sec'
logging.info(strRandSleepTime)

strThreadTimeOut = 'Orphan microservice kill time: ' + str(child_cond_time) + ' sec'
logging.info(strThreadTimeOut)

# Handling orphan microservice
if (random_sleep_time > child_cond_time):
    strCondChk = 'Rasing exception to avoid orphan microservice call!'
    logging.info(strCondChk)
    raise Exception

We tried to create a random wait time, which will produce a sudden network latency or mimicking potential network/time-out/connection issues. Also, we want to finish this thread to avoid any unwanted WhatsApp message trigger via Twilio-API.

And, when we want to test the parent Seat-Selection service through postman, it will show the following response –

Seat-Selection Microservice response

We must have noticed that since the boarding pass also captured the seat details; hence, we are addressing them together here.

For successful execution, we can see a similar message –

Seat-Selection service success status

However, if we want to handle this better, we can use “Circuit-Breaker.”

Let us understand what this is. Since our childhood, we all know that when there is an electricity fluctuation happens frequently, to prevent all the electrical appliances, we deploy this small device that will disconnect the power between our house’s primary grid with all the appliances at our home. It will prevent these devices from getting exposed.

The same concept has been implemented for Microservice architecture as well.

For Normal operation without any issue

The above diagram shows that using a circuit breaker will continue all its previous parent services in everyday activities without any issues.

Basic concept of the Circuit Breaker in Microservice

The above diagram shows that it will stop all the parent services if the child service is not responding based on the threshold count.

Circuit breaker with partially allow case

The above diagram shows that once there is a “STOP” event captured. It will wait for a few seconds before it will allow a few services to test whether the impacted service is ready to consume further requests or not. This way, it can control the threading bottleneck.

The next diagram will give us the distinct state of a “Circuit Breaker.”

States of a Circuit Breaker

The above diagram shows that using a circuit breaker will prevent all its previous parent services. It will maintain a threshold of failures & successful data points. It will allow services to invoke the impacted service either entirely, partially, or even completely block it depending on the situation.

Let us revise the code & review it together –

3. getSeatSelection ( This Microservice receives the inputs from its parent microservice & confirm the preferred seat-no selected by the passengers – using Circuit-Breaker. )

##############################################
#### Written By: SATYAKI DE               ####
#### Written On: 05-Oct-2020              ####
#### Modified On 05-Oct-2020              ####
####                                      ####
#### Objective: Scripts to choose the     ####
#### seat for the passenger.              ####
##############################################

import logging
import json
import time
import os
import ssl
import requests

import azure.functions as func

from circuitbreaker import CircuitBreaker

class BoardingPassCircuitBreaker(CircuitBreaker):
    FAILURE_THRESHOLD = 10
    RECOVERY_TIMEOUT = 60
    EXPECTED_EXCEPTION = TimeoutError

@BoardingPassCircuitBreaker()
def callService(inputJson, urlPost, retryNo, eventFlag, maxRetryNo, maxTimeOut):
    # Invoking getBoardingPass API
    try:
        # Bypassing SSL Authentication
        try:
            _create_unverified_https_context = ssl._create_unverified_context
        except AttributeError:
            # Legacy python that doesn't verify HTTPS certificates by default
            pass
        else:
            # Handle target enviornment that doesn't support HTTPS verification
            ssl._create_default_https_context = _create_unverified_https_context

        # Providing the url bearer token
        url = urlPost

        json_data = inputJson
        headers = {'content-type': "application/json", 'cache-control': "no-cache"}

        # Failed case retry
        retries = retryNo
        success = False

        return_description = 'N/A'
        statusVal = 'Failed'

        try:
            while not success:
                try:
                    # Getting response from web service
                    try:
                        strCheckMsg = 'Sending Payload - ' + str(retries)
                        logging.info(strCheckMsg)
                        
                        response = requests.request("POST", url, data=json_data, headers=headers, verify=False, timeout=maxTimeOut)

                        resp_dict = json.loads(response.text)

                        statusVal = resp_dict['status']
                        return_description = resp_dict['description']

                        logging.info(statusVal)

                    except Exception as e:
                        x = str(e)
                        logging.info(x)
                        success = False

                    if (eventFlag == 'Boarding-Pass'):
                        str_R = "Boarding-Pass Json Response:: " + str(response.text)
                    else:
                        str_R = "Invalid flag options "
                    
                    logging.info((str_R))

                    if len(response.text) > 80:
                        if str(response.status_code)[:1] == '2':
                            success = True
                        else:
                            wait = retries * 2
                            str_R1 = "retries Fail! Waiting " + str(wait) + " seconds and retrying!"
                            logging.info(str_R1)
                            time.sleep(wait)
                            retries += 1

                        # Checking maximum retries
                        if retries >= maxRetryNo:
                            success = True
                            raise  Exception
                    else:
                        if str(response.status_code)[:1] == '2':
                            success = True
                        else:
                            # Checking maximum entries
                            if retries >= maxRetryNo:
                                success = True
                                raise  Exception

                            retries += 1
                except:
                    strVal = 'Retrying - ' + str(retries)
                    logging.info(strVal)
                    # Checking maximum entries
                    if retries >= maxRetryNo:
                        success = True
                        raise  Exception

                    retries += 1

            selection_flag = 'Y'

            # Forming return JSON
            jsonRet = {
                "eventFlag": eventFlag,
                "status":statusVal,
                "description":return_description,
                "computeFlag":selection_flag
            }

            xval = json.dumps(jsonRet)

            return xval

        except (ConnectionError, TimeoutError, InterruptedError) as e:
            str_R8 = "Response from Server: " + str(response)
            logging.info(str_R8)

            str_R9 = "Sending payload to Webservice!"
            logging.info(str_R9)
            selection_flag = 'N'

            # Forming return JSON
            jsonRet = {
                "eventFlag": eventFlag,
                "status":"Lost",
                "description":"Timed-Out or Connection Error or any I/O issue preventing the process. We're working on this!",
                "computeFlag":selection_flag
            }

            xval = json.dumps(jsonRet)

            return xval

        except ValueError as e:
            x = str(e)
            logging.info(x)

            selection_flag = 'N'

            # Forming return JSON
            jsonRet = {
                "eventFlag": eventFlag,
                "status":"Failed",
                "description":"Please check the internal parameters compatibility with it's data-type!",
                "computeFlag":selection_flag
            }

            xval = json.dumps(jsonRet)

            return xval

        except Exception as e:
            x = str(e)
            logging.info(x)

            selection_flag = 'N'

            # Forming return JSON
            jsonRet = {
                "eventFlag": eventFlag,
                "status":"Accepted",
                "description":"Maximum rerties to the Boarding-Pass service is now reached. We're working offline to get that for you. Please rerty after 4 hours in case if you haven't received it or you can directly call customer care number!",
                "computeFlag":selection_flag
            }

            xval = json.dumps(jsonRet)

            return xval


    except Exception as e:
        x = str(e)
        logging.info(x)

        selection_flag = 'N'

        # Forming return JSON
        jsonRet = {
            "eventFlag": eventFlag,
            "status":"Failed",
            "description":"DevOps Engineer is looking into this. Please try after some time!",
            "computeFlag":selection_flag
        }

        xval = json.dumps(jsonRet)

        return xval

def main(req: func.HttpRequest) -> func.HttpResponse:
    logging.info('Invoking Seat-Selection generation service.')

    # Getting System Variable from local settings file
    strVal = 'Parameter Recieved :' + os.environ['maxRetry']
    
    logging.info(strVal)

    # Capturing parameters from local settings
    max_retries = int(os.environ['maxRetry'])
    c_url = os.environ['cUrlBP']

    # Printing input Payload
    str_val = 'Input Payload:: ' + str(req.get_json())
    logging.info(str_val)

    strMainCheck = str(req.get_json())

    # variable
    x_status = 'Success'

    if (strMainCheck != ''):

        # Capturing individual elements

        sourceLeg = req.params.get('sourceLeg')
        destinationLeg = req.params.get('destinationLeg')
        boardingClass = req.params.get('boardingClass')
        preferredSeatNo = req.params.get('preferredSeatNo')
        travelPassport = req.params.get('travelPassport')
        bookingNo = req.params.get('bookingNo')
        travelerEmail = req.params.get('travelerEmail')
        travelerMobile = req.params.get('travelerMobile')

        # Checking Individual Elements

        if not sourceLeg:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                sourceLeg = req_body.get('sourceLeg')

        if not destinationLeg:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                destinationLeg = req_body.get('destinationLeg')

        if not boardingClass:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                boardingClass = req_body.get('boardingClass')

        if not preferredSeatNo:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                preferredSeatNo = req_body.get('preferredSeatNo')

        if not travelPassport:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                travelPassport = req_body.get('travelPassport')
        
        if not bookingNo:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                bookingNo = req_body.get('bookingNo')

        if not travelerEmail:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                travelerEmail = req_body.get('travelerEmail')

        if not travelerMobile:
            try:
                req_body = req.get_json()
            except ValueError:
                pass
            else:
                travelerMobile = req_body.get('travelerMobile')


        if (
            (sourceLeg != '') & (destinationLeg != '') & (boardingClass != '') & 
            (preferredSeatNo != '') & (travelPassport != '') & (bookingNo != '') &
            ((travelerEmail != '') | (travelerMobile != ''))
            ):

            x_description = "Seat-Selection Successfully Processed!"

            # Preparing Payload for boarding-pass
            json_str_boarding = {
                "sourceLeg": sourceLeg,
                "destinationLeg": destinationLeg,
                "boardingClass": boardingClass,
                "confirmedSeatNo": preferredSeatNo,
                "travelPassport": travelPassport,
                "bookingNo": bookingNo,
                "travelerEmail": travelerEmail,
                "travelerMobile": travelerMobile
                        }

            jsonBoardingInput = json.dumps(json_str_boarding)

            eventFlag = 'Boarding-Pass'
            retryNoBase = 1
            maxTimeOut = 2.5

            # Invoking getBoardingPass API
            microCallsBP = callService(jsonBoardingInput, c_url, retryNoBase, eventFlag, max_retries, maxTimeOut)
            
            # Extracting seat_selection_flag
            resp_dict = json.loads(microCallsBP)
            x_status = resp_dict['status']
            x_description = resp_dict['description']

            # Capturing description value based on the logic
            logging.info(x_description)

            # Formatting return Payload
            json_str = {
                "description": x_description,
                "status": x_status,
                "sourceLeg": sourceLeg,
                "destinationLeg": destinationLeg,
                "boardingClass": boardingClass,
                "confirmedSeatNo": preferredSeatNo,
                "travelPassport": travelPassport,
                "bookingNo": bookingNo,
                "travelerEmail": travelerEmail,
                "travelerMobile": travelerMobile
                        }

            xval = json.dumps(json_str)

            return func.HttpResponse(xval, status_code=200)
        else:
            json_str = {
                "description": "Missing mandatory Email or Phone Number!",
                "status": "Failed",
                "sourceLeg": sourceLeg,
                "destinationLeg": destinationLeg,
                "boardingClass": boardingClass,
                "confirmedSeatNo": preferredSeatNo,
                "travelPassport": travelPassport,
                "bookingNo": bookingNo,
                "travelerEmail": travelerEmail,
                "travelerMobile": travelerMobile
            }

            xval = json.dumps(json_str)
            return func.HttpResponse(xval,status_code=200)
    else:
        json_str = {
            "description": "Missing entire payload!",
            "status": "Failed",
            "sourceLeg": "N/A",
            "destinationLeg": "N/A",
            "boardingClass": "N/A",
            "confirmedSeatNo": "N/A",
            "travelPassport": "N/A",
            "bookingNo": "N/A",
            "travelerEmail": "N/A",
            "travelerMobile": "N/A"
        }

        xval = json.dumps(json_str)
        return func.HttpResponse(xval,status_code=200)

Let us review the key snippets –

from circuitbreaker import CircuitBreaker

class BoardingPassCircuitBreaker(CircuitBreaker):
    FAILURE_THRESHOLD = 10
    RECOVERY_TIMEOUT = 60
    EXPECTED_EXCEPTION = TimeoutError

@BoardingPassCircuitBreaker()
def callService(inputJson, urlPost, retryNo, eventFlag, maxRetryNo, maxTimeOut):
    # Invoking getBoardingPass API
    try:
	    <Code>

            # Forming return JSON
            jsonRet = {
                "eventFlag": eventFlag,
                "status":statusVal,
                "description":return_description,
                "computeFlag":selection_flag
            }

            xval = json.dumps(jsonRet)

            return xval


    except Exception as e:
        x = str(e)
        logging.info(x)

        selection_flag = 'N'

        # Forming return JSON
        jsonRet = {
            "eventFlag": eventFlag,
            "status":"Failed",
            "description":"DevOps Engineer is looking into this. Please try after some time!",
            "computeFlag":selection_flag
        }

        xval = json.dumps(jsonRet)

        return xval

We have put the microservice call into a function & then mark it as a circuit-breaker method. Also, we have enforced some custom settings in our main circuit-breaker class as well.

Moreover, here are some of our main API sample outputs, i.e., “getOnBoarding,” depending on the child-services’ availability.

Please click the above image gallery to see all the combinations of responses.

Finally, here is the WhatsApp message if that call successful. We have used Twilio API to exchange the text.

Boarding Pass delivery to Passenger WhatsApp No

Please find the dependent package from requirements.txt ->

azure-functions==1.4.0
certifi==2020.6.20
chardet==3.0.4
circuitbreaker==1.3.1
idna==2.10
numpy==1.19.2
pandas==1.1.2
PyJWT==1.7.1
python-dateutil==2.8.1
pytz==2020.1
requests==2.24.0
six==1.15.0
twilio==6.45.4
urllib3==1.25.10

So, finally, we have done it.

I’ll bring some more exciting topic in the coming days from the Python verse.

Till then, Happy Avenging! 😀

Note: All the data & scenario posted here are representational data & scenarios & available over the internet & for educational purpose only.

One thought on “Managing mesh-APIs a few best practices including circuit-breaker & more.