Building an AI Chatbot using a Regular Expression Engine

• Download the Verbot SDK - 627.4 KB

Introduction

This article describes how to build an Artificial Intelligence (AI) Chatterbot using an open source, Regular Expression-based natural language processing engine called Verbot. Chatterbot technology is not new, but has been growing in popularity and usage as people experiment with new ways of interacting with computers. Chatterbots, Animated Characters, Speech Recognition and Voice Synthesis are being used to facilitate new human–computer interaction (HCI) channels.

Background

The original chatterbot, ELIZA was developed by Joseph Weizenbaum in 1966. ELIZA was a virtual psychotherapist that would attempt to engage the user by staying on topic and sounding concerned with his/her problems. In 1994, Michael Mauldin -- the creator of the first Verbot, Sylvie -- coined the term "ChatterBot." Since then, Verbots have evolved and matured into the current Verbot 4 version created by Conversive. In 2006, Conversive released the Verbot SDK as open source to allow people to easily create their own 'bots and embed them in their own applications. This article describes some of the basics behind the Verbot Engine SDK and how the SDK can be integrated into another application.

How the Verbot works

Before delving into code, I thought it would be useful to explain at a high level how the Verbot SDK works. All chatterbots have an idea of a "KnowledgeBase" or script, which tells the 'bot what keywords or phrases to recognize. The corresponding output is then sent to the user when a match is found. Verbot KnowledgeBases are made up of a collection of rules that are the fundamental building blocks of Verbot conversations. Rules must have one or more Inputs, which recognize what the user says. They must also have one or more Outputs, which are what the 'bot responds when an input in the rule is matched. When the Verbot Engine loads a KnowledgeBase file, it turns the simple text inputs found in the script into regular expressions that can then be used as the basis for matching.

For example, let's say we have an input which is:

What is your name?

The resulting regular expression generated by the engine looks like this:

^(|.*?\b|.*?\s)What\b.+?\bis\b.+?\byour\b.+?\bname(|\b.*?|\s.*?)$

This may be a bit daunting if you are unfamiliar with Regular Expression syntax. Even if you are familiar, it may still seem a bit cryptic. Basically, the engine has put wildcards between words, as well as at the start and end of the phrase. This regular expression will match to, "So, what the heck is your name?" but not to, "What's your name?" To match to this, we could use the Synonym feature in Verbots, which you can learn more about at their website along with other, more advanced scripting topics. Also, there is a good Regular Expression syntax reference over at dotnetcoders if you'd like to learn more about regex.

Verbot KnowledgeBases (.vkb extension) are XML files that define the rules, inputs and outputs of the KnowledgeBase. They may also reference other Verbot 4 file types such as Synonyms, Replacement Profiles and Code Modules. Verbots allows you to "compile" your VKB files into a more compact binary "Compiled KnowledgeBase" (.ckb extension) file format.

Testing the application

Execute the VerbotWindowsApplicationSample.exe file found within the VerbotWindowsApplicationSample\bin\Release folder. Once the application is running, choose the File -> Load... menu item and browse to the VerbotWindowsApplicationSample\Resources\sample.ckb file to open the sample KnowledgeBase. Once the file is loaded into the engine, you may interact with the 'bot. Try typing "notepad" to test out one of the rules included in the sample KnowledgeBase.

Notice how the program has started Windows Notepad. This is behavior coded into the Sample application to parse commands in the script. I'll talk more about this later.

Using the code

To use the Verbot Engine SDK inside your own application, simply include the Verbot4Library reference in your project, create an instance of the Verbot4Engine and call either AddCompiledKnowledgeBase or AddKnowledgeBase to load your 'bot's knowledge. To interact with the 'bot when user input comes in, create a State object to uniquely identify your user. Add the paths to the KnowledgeBases you want your user to interact with. Then call the GetReply method on the Verbot4Engine object to find a match in the loaded script(s).

Let's look at the getReply() function in our VerbotWinApp.cs file:

private void getReply()
{
    string stInput = this.inputTextBox.Text.Trim();
    this.inputTextBox.Text = "";
    Reply reply = this.verbot.GetReply(stInput, this.state);
    if(reply != null)
    {
        this.outputTextBox.Text = reply.Text;
        this.parseEmbeddedOutputCommands(reply.AgentText);
        this.runProgram(reply.Cmd);
    }
    else
        this.outputTextBox.Text = "No reply found.";
}

This method takes the text in the input text box and sends it to the engine's GetReply method. It gets a Reply object back, which we can then process. The Reply object has a member "Text" that we will just put in the output box. It also has an "AgentText" member, which is a version of the Text meant to be sent to a Text-To-Speech (TTS) engine. It has a "Cmd" (Command) member, too, that we can use for special processing as necessary.

Finally, take a look at the parseEmbeddedOutputCommands and runEmbeddedOutputCommand methods. These methods do custom processing based on our requirements. This is where the "run" command opens Notepad, as I mentioned above.

Depending on how your own application works, you can make customizations to how the Reply object is handled. For example, in an online 'bot engine you may want to process URLs by opening a popup window, etc.

Creating knowledge

There are some freely available tools to help you create a Verbot KnowledgeBase. Verbot GPL Editor and VCompiler are a couple of options to get you started. You can also use the Verbot Editor, which costs $10 to unlock with a registration code. If you'd like to create your own KnowledgeBase creation tool, the code is pretty straightforward. You may also want to look at the source code for the Verbot GPL Editor I mentioned above as a good reference. Here is a summary of the steps required to create a KnowledgeBase in code:

1. First, create an instance of the KnowledgeBase class:

   KnowledgeBase kb = new KnowledgeBase();

2. Then we'll want to create a rule to add to our KB:

   Rule r = kb.AddRule();

3. Finally, for each rule you create you'll want to have at least one input and one output. We could manually create and add inputs and outputs or we could simply call the AddInput and AddOutput methods in the rule class:

   r.AddInput("Input Text", "");
   r.AddOutput("Output Text", "", "");

Now we've got a KnowledgeBase object with one rule that has one input and one output. Of course, in a real application we wouldn't want to hard-code the "Input Text" and "Output Text" fields and would probably want to allow the user to add these themselves, but you get the idea. Next, we probably want to save our KB to a file. To do this, we need an instance of the XMLToolbox class. Then we simply call the SaveXML method to save the file:

XMLToolbox xmlToolbox = new XMLToolbox(typeof(KnowledgeBase));
xmlToolbox.SaveXML(kb, @"C:\mykbpath.vkb");

Then we want to be able to load our KnowledgeBase into the Verbot Engine and be able to interact with it. Here is the code to do that:

Verbot4Engine engine = new Verbot4Engine();
KnowledgeBaseItem kbi = new KnowledgeBaseItem();
kbi.Fullpath = @"C:\";
kbi.Filename = "mykbpath.vkb";
engine.AddKnowledgeBase(kb, kbi);
State state = new State();
state.CurrentKBs.Add(@"C:\mykbpath.vkb");

Finally, to interact with the loaded knowledge bases just call the GetReply method on the Engine, as discussed above.

Further reading

If you are interested in learning more about Verbots, visit the Verbot community forums or browse Verbots wiki to learn more about scripting your own Verbot.

History

• 22 March, 2007 -- First draft of article.
• 28 March, 2007 -- Updated article for new version of SDK 4.1.3.2, which includes a simple console application as a sample and a new KnowledgeBase method, AddRule, which returns the new rule. Also, small changes in article instructions were made.
• 21 June, 2007 -- Article edited and moved to the main CodeProject.com article base.