Integrating Azure QnA Maker Service as a Bot Middleware

Recently I have been working with Azure QnA maker cognitive service and was integrating it with bot framework built using SDK v4 and clearly had two choices as an implementation approach

• Implement it as a regular workflow in bot 
• Implement it as a bot middleware

This article assumes that you understand bot framework and you are aware of few essential concepts around e.g. bot services, configuration and middleware etc. If you are new to it, then it is highly recommended that you go through this documentation and learn it before proceeding further.

In short, the middleware is a component which sits in between the channel adapter and bot and can see and process each request sent to the bot and any response sent by the bot to the user. 

In a nutshell, the channel adapter is a component which is responsible for converting channel data into a JSON format which can be understood by the bot framework.

As we can see in the image above that middleware sit one after the another in the execution pipeline before the message sent to the bot received by the bot and then gets executed in reverse order when response is being sent to the user by bot and hence the registration sequence of each middleware matters. 

One of the practical examples could be 

Now let’s get to the point and see how we can integrate QnA maker cognitive service as middleware. Why does it make sense to create it as a middleware?

Well, one of the reasons for this decision was because we first wanted to try and get response to any user message by searching inside the QnA repository and if it nothing is fetched then bot core logic can take care of the sent message and run business rules on it.

You can go through the process of creating QnA maker service in Azure portal here and understand how to train it by feeding data, your sources can be public URL of your documentation, tsv files, or excel sheets containing FAQs.

To work with QnA service and to use it inside the bot you will four things

• Your knowledge base Id
• QnA service endpoint key / access key
• Azure web app host url (gets provisioned when QnA service is created)
• NuGet - Microsoft.Bot.Builder.Ai.QnA - https://www.nuget.org/packages/Microsoft.Bot.Builder.AI.QnA/

All these can be a part of your .bot file can be initiated at the time of your bot startup in startup.cs, sharing a sample of .bot file with QnA service configuration.

and Startup.cs will look like this

Please note that the code above is just for conceptual reference and is referred from the available samples here. You can find definition of the BotServices.cs on same link.

And now with this, since we have configured the bot with QnA maker service, lets move on to middleware.


As you can see that the source code is quite self-explanatory – all we are doing is implementing the OnTurnAsync method of the IMiddleware interface and simply calling QnA service to fetch results.

Note that this implementation has been kept simple for reference, but you can customize it e.g. processing retrieved response before handing it over the user, log it etc.

Also note that currently it checks for the scores of retrieved results from QnA service and tries to build the top response. You can further configure the 50% threshold value as per your need.

If we receive any response from the QnA service, we are ending the conversation there and handing over the response directly back to user without invoking further components sitting in the pipeline in this sample.

Now the last part is to make bot aware of this middleware by registering it in Startup.cs


And that’s it, now your QnA service is set using a bot middleware.
Hope this helps someone.

Building Proactive Messaging Bot using Bot framework SDK v4

I have been working with recently released bot framework from last few weeks and one of the interesting requirements I came across was to build a proactive bot which can send messages to users on its own.

Now this sounds like a very straight forward and quite common requirement as possibility of having this capability in bots opens a gateway to number of opportunities in an enterprise environment e.g. sharing live information analytics to power users, sharing caution / warnings to system administrators based on past data etc. however there is a serious limited documentation available on achieving these real-world scenarios specifically with Bot framework SDK v4.

In this post, I am going to share the experience I had while developing a proactive bot and challenges which I faced, approach I took by referring several links and will also share what I learned.

Hitting the road block

To start with, at the time of writing this article - the only documentation and an official sample available from Microsoft is as mentioned below. Please go through these links and understand the essentials involved in getting a proactive bot design working.

• Send proactive messages
• Proactive-messages – sample available on Git

Coming back to the original requirement we had which was inline with what MSDN documentation says i.e.

"If the user has previously asked the bot to monitor the price of a product, the bot can alert the user if the price of the product has dropped by 20%"

But the big question was - how exactly do you do that?

Understanding basics

Now to understand this, lets get into the technical details and understand what it essentially means to send a proactive message to the end user from bot perspective.

Proactive conversation is nothing but the event or a message that bot sends to the user and there is one fundamental need here that the user should have at least initiated a conversation with the bot first else it does not make sense that users starts receiving messages from bot which they do not know! 

Now coming to the next point – as we know that overall bot framework is built around the idea of web api and is stateless, hence all the conversations or dialogs or messages which we send to the bot are essentially listened by one web api endpoint (if you have noticed, there is always this default route and an endpoint https://localhost:1234/api/messages) – whenever this api receives any message – it is then processed and response is sent back to the user via specific channel. (channel is just a medium using which user is talking to the bot e.g. teams, direct line or skype). 

The requirement which I was working with needed some trigger from external system to the bot endpoint so that bot can process the incoming signal and then send out a conversation to a specific user.

Let’s understand this using a quick diagram

Note that - to send the conversation proactively to the user – the bot MUST know who the user is and what is his / her conversation reference otherwise it won’t be able to initiate the conversation with the user. 

Approach

Content below is heavily inspired by this github thread and a big shout out to all the contributors on that thread – you guys are awesome!

Bringing it all together:

These are the basic steps we integrated to get the proactive messaging working

• Record conversation reference of each user / desired users
• Used Azure table storage to store these records 
• Introduce api to fetch stored record from storage and generate ProactiveMessage
• Introduce / Integrate ProactiveController from this post
• Trigger post call to ProactiveController’s method with payload generated from step 3 and send a message from bot to a specified user

Let’s see each step-in detail

Step 1 and 2:

Because the pre-requisite is bot should know the conversation reference before it can send the message by its own, it becomes essential to have the conversation references stored somewhere.

Since we do not really need a relational data structure here, so we can store it azure table storage, but you can choose to store in relational database table too. 

How do we store?

We can either write a custom middleware and detect if the entry for the user exists in the underlying storage, if it does not – meaning it is a new conversation and we record it. one big problem with this approach could be – since it is a middleware – it would be invoked every time for each message and might impact performance. 

Thankfully, in our bot since we have integrated Azure AD authentication – we knew exact place where we can write this storing logic i.e. whenever user gets authenticated. 

This is how my UserEntity look which I am storing in the table storage

Instance of this class needs to be initiated from the incoming first user request which you would find in the OnTurnAsync method

Step 3:

Because we have now stored the user entity (conversation reference) in the table storage – we will now need the api which can query the storage based on inputs provided.

E.g. if I am storing user name in FromId property, then I would need the api to generate Activity Payload for that specific user so that we can send it further to the ProactiveController which takes care of beginning the proactive conversation with that user.

I have added a controller in the same bot project which returns the collection of ProactiveMessages
Here my definition of ProactiveMessage


And ActivityGeneratorController in short


And here is the sample output it generates


IMPORTANT NOTE:
This took a while for me to understand and hence sharing. The botId and needs to be the Microsoft App Id of your bot and then only it works.

Step 4 and 5:

Finally, you should integrate the ProacticeController in your bot project. 

I am not going to share that source code here again because there is already a git project for it.

Validation:

Once this set up is completed, and if you have managed to generate the activity payload correctly by recording correct user conversation references, you can try sending the HttpPost request to the ProactiveController externally from any request generator like Postman and you should see the proactive message with your text appear in conversation of the user to which it was sent. Note that in above Json example, the message with text “Test Message “will appear in user’s team channel.

Update

The approach mentioned above stopped working in specific scenarios e.g. when the user to which message needs to be sent becomes idle on channel then proactive messaging started throwing exception below

Microsoft.Bot.Schema.ErrorResponseException: Operation returned an invalid status code 'Unauthorized'
   at Microsoft.Bot.Connector.Conversations.ReplyToActivityWithHttpMessagesAsync(String conversationId, String activityId, Activity activity, Dictionary`2 customHeaders, CancellationToken cancellationToken)
   at Microsoft.Bot.Connector.ConversationsExtensions.ReplyToActivityAsync(IConversations operations, String conversationId, String activityId, Activity activity, CancellationToken cancellationToken)
   at Microsoft.Bot.Builder.BotFrameworkAdapter.SendActivitiesAsync(ITurnContext turnContext, Activity[] activities, CancellationToken cancellationToken)
   at Microsoft.Bot.Builder.TurnContext.<>c__DisplayClass22_0.<<SendActivitiesAsync>g__SendActivitiesThroughAdapter|1>d.MoveNext()

and this puzzled us, again there is a lack of documentation and information is scattered around different links and blogs. We came across one helpful link which gave some hints and then we had to make changes in out ProactiveMessagingController logic which now looks like this



Note that we are now creating channel account and initialing an instance of MicrosoftAppCredentials with bot's app Id and secret (typically you would find this in .bot file in your solution, if not you can always get it from bot registration portal) and then with the help of ConnectorClient we are not beginning conversations with specific user.

Hope this helps someone looking to build the proactive messaging using bot framework SDK v4 and with this one can trigger targeted proactive messages externally to specific user with notifying them with relevant message.

Implementing database per tenant strategy in bots using Azure bot services and SDK v4

Recently I have been working with building the bot with latest bot framework SDK i.e. v4 and the aim was to provision the bot as a SaaS. Meaning that, we already had dedicated SQL Azure database instances provisioned for each tenant (which are also consumed by other line of business applications) and there was need to create the bot which will be hosted with Azure bot service and authenticated by Azure Active Directory. Whenever user logs in to the bot, based on the identity of logged in user the bot should dynamically redirect all the subsequent data read and write requests to correct SQL Azure database instance of the user.

This article does not focus on writing an end to end bot covering scenario above (but can be a good candidate for later) and makes some assumptions that you already have some knowledge of latest bot framework and are familiar with fundamentals of it along with .NET core and Entity Framework core. If you are new to it, then it is highly recommended that you go through documentation here and then continue reading the further part of this article.

The basic architecture and overall Eco-system looks like this

Note that the sole purpose of sharing the above diagram is to give you a high-level overview of the solution and to make you aware that where exactly bot service is coming in to the picture. You can choose to ignore the data preparation and reporting part which is at the right and involves ADF and Power BI.

To keep the content of this article focused on our scenario, we will not go in to the details of how bot can be configured to use Azure AD for authentication and blob storage for state management etc. You can find the documentation to do it on this and this link.

One quick point to mention here is, the Azure AD app needs to be configured as multi-tenant app so that users from different active directories would be able to get themselves authenticated.

Now coming back to the problem of redirecting requests to right tenant based on the user identity, let’s see how that can be done.

Typically, when we want to use the EF data context object inside the ASPNET core application, we end up taking help of DI and inject it in Startup.cs which looks something like this


But this does not help here, Why? Is because at the time of the application startup – you do not know which database we want to point to.

DbContext object should be initialized at the run time once we know that who is the user and where is the relevant database for that user.

Now, to know where to send data request of users – we need to have some sort of mapping present between users logging in to bot and respective database connection strings. This mapping can be done in number of ways e.g. maintain mapping of user’s domain and connection strings or maintain mapping of user’s tenant id and connection strings.

You can choose your own store to have these mappings provisioned i.e. store it either on central SQL azure database table or maintain it in XML, JSON or in configurations or in Azure key vault.

There is an excellent documentation available by Gunnar Peipman explaining this scenario, you can have a look at it to understand the details of how it uses json file to initialize different mappings

So functional flow looks like this

Runtime Initialization of the DbContext object can be easily achieved by overriding OnConfiguring method of the DbContext class.


With this approach, there is not really a necessity to inject the DbContext at the time of application startup and instead you can initialize DbContext explicitly whenever you need. Note that every time you request or create your DbContext object – the OnConfiguring method will be called.

Extending the model

How do I get the connection string at the run time?

The answer is, you can create your own connection string provider which will return you connection string based on user and will look up relevant connection string either in Sql database, or in configuration or in azure key vault.

For our specific scenario, since our bot is AD authenticated – once the user is logged on, we know the tenant id of the user. We store that tenant id in custom class i.e. UserProfile and store it in the bot user state so that it can be accessed during any conversation.

As a mapping between tenant id of the user and actual connection string, we do use the Azure key vault to store connection string value as a secret. You can refer how to configure azure key vault connected services aspnet core in this article - Add Key Vault to your web application by using Visual Studio Connected Services

The reason tenant connection string provider comes handy when tomorrow you decide to maintain mappings in some external secure service other than key vault in that case all you would need to do is, replace the implementation of tenant provider and that’ll be all.

Here is how tenant connection string provider can look (fetching connection strings from Azure key vault)

Introducing DbContextFactory

It always is a good approach to introduce the factory pattern to achieve the additional isolation between the requester and object creator.


Note that how we have created our own implementation of CreateDbContext method instead of implementing the default method which only takes string array as input.

Also note that how we have consumed the ConfigurationTenantProvider which implements TenantProvider’s GetTenant method and is configurable. We could have injected the ITenantProvider in factory but for this example, just wanted to keep the setup simple and hence initialized it directly.

Let’s understand it with the help of a block diagram

Startup configurations

The last part would be to make DbContextFactory available to all the dialogs i.e. by injecting it to the container.


and since we now have registered the DbContextFactory to the DI container, you should easily be able to access it in any bot dialog and should be able to create your DbContext dynamically whenever needed.

Summary

The latest bot framework and EF core is quite flexible, approach explained above is one the way to make your db context dynamic and there could be multiple ways to do the same. Readers are encouraged to refer it as an option and customize it further as per their needs.