Writing a basic chatbot using Microsoft Botbuilder and LUIS

While we're building our own custom chatbot framework here at Craftworkz, we of course like to keep an eye out for the competition. One of the latest new kids on the block, eager to get a piece of the pie while it's still hot, is Microsoft. Microsoft launched its Language parsing API LUIS (Language Understanding Intelligent Service) at the end of last year, and the accompanying BotBuilder SDK is only a few months old. And while both are still very much in development, we decided to check it out already and see how it compares to the competition.

First of all, it’s important to understand the distinction between LUIS and the BotBuilder SDK. The first is used exclusively to allow your bots to process natural language sentences in an intelligent way in order to find out what the user is actually trying to say. The BotBuilder uses these parsed sentences to return a valid answer to the user. In fact, it is perfectly possible to build a bot using the framework alone, let’s start there.

BotBuilder SDK

Getting started using the SDK is quite easy: you can either use Node.js or .NET (this is Microsoft after all), or work exclusively through a REST-based API. Forced to say goodbye to my beloved Python, I went for the Node.js option since that’s what I’m most familiar with.

After installing the BotBuilder SDK, the first steps are pretty straightforward: the following code snippet shows how to build a simple hello world robot:

Disregarding the formatting, the code contains four lines in total: the first imports the BotBuilder package, the second allows you to use the console to interact with the bot, the third instantiates the bot itself and the last line contains the dialog info. I’ll be using a slightly more advanced version allowing me to use the Bot Framework Emulator to interact with the Chatbot. Let’s give it a try.

The result, as you can see, is not very impressive. Whatever I say to the chatbot, it always replies with the same sentence. In order to create an intelligent chatbot, two improvements are required:
- The bot needs to be able to remember data, both from the user and the conversation it is currently having.
- The bot must be able to understand what the user is saying.

We’ll add both these improvements in version 2.0 of our MicrosoftBot. The first three lines of our program remain the same, but now we’ll expand the dialog part of our code to include a very important concept: Intents. Intents determine what your user is trying to do with your bot, for example, making an appointment or getting some information about a certain subject.

No matter the framework you’re using, determining the intent of the user is the main objective when building a chatbot (that, and extracting entities, but more on that later). So, let’s give our brand spanking new bot a spin, and see how it performs.

Wow, I have now built a chatbot that echoes whatever compliment I tell it to. Safe to say, the business potential of this bot for CEO’s and politicians is huge!

You can see the two improvements I talked about earlier at work here. First, the bot now remembers the sentences I tell it. Second, I am able to interact with the bot by telling it to say something else. Both of these concepts are crucial when creating an interactive chatbot.

By now you might be wondering why we even need a language parsing engine like LUIS in the first place. Well, let’s assume I’ve gotten quite attached to my little compliment bot and feel like it deserves to be treated with a little more respect. From now on I will try to be more polite when I’m interacting with my favorite bot, let’s see what happens.

Uh oh! It seems my bot does not seem to respond well to my new found sense of politeness.
This is because I programmed the intent handler to only respond to a specific sentence. To be fair, you could enlist the help of your local regex wizard to improve this by triggering the intent when certain keywords are present (e.g. “say, something else), but even the almighty power of regex wizardry can only get you so far.

You see, human language is very hard. There are a million different ways to say the same thing (e.g. say something else, change your sentence, use a different phrase,…), and parsing these sentences require advanced algorithms that have only been usable for a couple of years. Let’s move on to part two, and I will show you how it works.

LUIS

Alright, this is where the magic happens! The first thing to understand is that LUIS is not a chatbot-, but a language parsing engine. This means that you’ll still be writing chatbots as I’ve shown in the previous part, only now LUIS will extract the intent of the user in a more intelligent manner.

The way LUIS works is that it allows you to define the possible intents of your user, and then you can train LUIS to recognize this intent by providing a few sample phrases. LUIS will then determine the key elements in these sentences, allowing it to recognize the intent even if it hasn’t seen the exact sentence before. Allow me to demonstrate.

As you can see on the left, I have defined three possible intents to demonstrate the concepts used by LUIS. I have provided about ten sample sentences for each intent.
Starting with the most basic example “Info-General”, this intent is triggered when the user wants to know more about the bot. On the right, I have asked our bot to tell me something about itself in a new way, and it has correctly identified my intent.

By now you might have noticed another important concept on the left called ‘Entities’. Imagine you want to build a chatbot that allows users to find wikipedia explanations of certain concepts. Right now, the only possible way to do this would be to define an intent for every possible concept your users want an explanation for, such as wikipedia-apples, wikipedia-DavidBowie, wikipedia-NewtonsLaws, etc. It is clear that this is would be impossible to do, and that’s why we’re introducing Entities, the variables of the language parsing world. Once again, I will demonstrate using an example.

This time I trained the Info-Subject intent by asking LUIS about a variety of different subjects in different ways. Each time, I tagged the subject of my inquiry as an Entity-Info entity.
Now LUIS can recognize when I’m asking about a specific subject, and can even identify what it is I’m asking about.

The combination of Entities makes for an incredibly powerful combination. LUIS even offers some prebuilt entities like datetime, which I’m using for my final example. These prebuilt entities are processed automatically by LUIS before being returned to the ChatBot framework saving the developer a lot of parse work. For example, if I say “in 5 days”, LUIS will automatically return the actual date to the BotBuilder.

In the final example, I will combine all the concepts I have shown previously to implement a Make-Appointment intent. When making an appointment, each appointment should have three entities: a target (who you’re making an appointment with), a subject and a time. Defining these entities is just as easy as the previous example. Once again, I trained the Make-Appointment with about ten sentences.

Again, LUIS had no problems identifying both the intent of the user and the relevant entities. These entity values are then sent to the BotBuilder SDK program, where they can be processed as you would in any other regular program.
Now, you might be wondering: “Well that’s all very impressive (thank you), but what happens when a user forgets to input one of these values? Do I need to create an intent for each of these cases?”

That is an excellent question dear anonymous reader, and the answer is “No you don’t”. Leveraging the flexibility of LUIS and the power of the BotBuilder SDK, you are able to interactively prompt the user for the missing variables. I won’t go into the technical details, but I will demonstrate by interacting with the final version of the Microsoft Chatbot.

There you have it! All that’s left for me to do is upload my BotBuilder code to Azure, publish my bot on all relevant channels and watch the appointments flow in!

Conclusions

Having worked with Microsoft Botbuilder and LUIS, I will now try to give you an overview of the positive- and negative aspects of both platforms, specifically compared to our own custom platform and Watson conversations.

The Good
- Powerful pre-built entities
- Clear LUIS interface
- Easy to prompt users for missing entities

The Bad
- Unclear- and badly documented BotBuilder API
- External API support not built-in
- Some advertised features of LUIS not present (yet)
- No dutch localisation

The Ugly
- Both platforms are still in beta, meaning potential application-breaking changes are possible at any time.

All in all, LUIS is quite impressive, especially considering the notoriety of Microsoft regarding buggy software and less-than-ideal user interaction. The coming weeks we’ll be expanding our framework to incorporate some of the strengths of LUIS and the BotBuilder into our own system.

However, we at Craftworkz pride ourselves in our capacity to make use of a wide array of technologies. If you are already running a Microsoft-based stack, we can also make use of Microsoft services like LUIS and Azure to work around your needs, as we have demonstrated here.