Click here to Skip to main content
Click here to Skip to main content

Open Dyno Real Time Control System - Part 3

, 18 Apr 2014
Rate this:
Please Sign up or sign in to vote.
The Open Dyno Real Time Controller

Introduction

Download OpenDynoRT source

This is the third article in the series about Open Dyno. The previous two articles showed how some of the GUI applications work and how to input configuration information. The concept of points, alarms, tests, and scripting were developed. This article will show how the configuration information for the points, alarms, and scripts are used by the real time application. The real time application is a multithreaded console application written in C++.

The main goal of the real time application is to collect and process data on a timely schedule. The Open Dyno control system tries to maintain a sample time of 100Hz or 10ms between samples. This is illustrated below. The green lines are the desired sample times and the blue lines are actual sample times. The difference between the desired time and actual time is called jitter. The amount of jitter in a control system is hardware dependent and can affect the ability to optimally control a device. Jitter is caused by delays in the time that the operating systems services an interrupt. Jitter can be minimized by using a data acquisition device that supports hardware timed samples such as a PCI or PCIe card or a real time communication system such as EtherCAT. For some control systems jitter isn't an issue and even missing a sample will not cause a failure. These are called soft real time. If jitter is a big issue and missing a sample could cause failure, the system requires a hard real time operating system such as RT Linux or QNX.

The time between the samples is where all of the processing is done and is the main focus of this article. During this time the input data is collected, algorithms are calculated and then a response with all the outputs is sent back to the data aquisiition device. At the time of sampling, the input data values are collected and the new output values are set. All of the processing must be completed before the next sample so that the output values are ready to be written. It isn't an issue if the processing takes a variable amount of time as long as it's complete before the next sample. Some ways to minimize the variability in processing time are: not to allocate memory or access a disk device during the processing step. Using threads for background processing, communication, and data logging eases the amount of work required in the main loop.

Background

At the start of development the decision was made to use the POCO C++ libraries to allow the application to run on both Windows and Linux with very little modification. POCO is a cross platform library of C++ code that provides low level abstractions for many operating systems. For example, it provides Sockets, Files and Paths, Threads, and many other useful classes. I primarily use the Windows version as a simulator so that I can test the Open Dyno GUI applications. I've only used a relatively small portion of POCO. It has many more features that could be used to extend the real time application even further.

The real time application is designed to run without any keyboard interaction. All of the commands come remotely from an Ethernet socket connection. This is the typical scenario found in industrial controls where an HMI and a PLC are used to control a process. The HMI talks to the PLC over some form of communication bus. In the case of Open Dyno, the GUI takes the place of the HMI and communicates with the real time console application over Ethernet. One of the nice features of separating the real time code is that it can easily run on an embedded device.

Open Dyno is a second generation control system. In the original real time application there were a couple of threads that communicated directly with hardware devices. In this revision the decision was made to exclude any knowledge of I/O devices from the core application. This was done to remove any specific hardware requirements and also so that the simulator could run on any machine without the need for I/O devices. I use the term simulator to refer to the non real time Windows version of the application. The simulator is a very good tool for training and testing algorithms. A new user can try out all of the applications without the fear of damaging any real equipment.

There were many possible ways to get I/O data from devices into the application. It could be done using plugins or dlls, shared memory, or some form of inter process communication. The final decision was to use Ethernet UDP Sockets. This provides the most flexibility and the performance has proved good enough for the 100Hz requirement. This requires another application to act as middle tier to collect the I/O data and transfer it to the real time application. A very simple binary packet called an Open Dyno Message was developed that can be created and sent from virtually any programming language. For my current project, LabVIEW Real Time is used for collecting data from data acquisition devices. LabVIEW code was also developed to receive Open Dyno Messages to update output data. There is no requirement to use LabVIEW and most of the original testing was done using a simple C# application. On Windows I am able to receive about 120,000 Open Dyno Messages a second with 256 points in each packet from C# (each point is an 8 byte double). The nice thing about UDP is that it doesn't require a connection. This enables many applications to send data to the same port.

Application Architecture

Poco provides a skeleton template for a console application. The template provides support for command line arguments and configuration files. The application template also provides initialization and un-initialize routines for starting and stopping the application. Rather than re-invent all of this code it was simply extended for this particular application. During the initialization routine all of the data structures and threads are created. These threads stay active until the un-initialize routine. The block diagram below illustrates this.

In the previous articles the concept of points, alarms, and scripts were all discussed. It was shown how to edit them using the GUI and how there are stored in the database. The information from the database is used in the real time application to create point, alarm and script objects that use this data for configuration. There is one global variable in the application called dataPoints. This is an std::map<string, double> that is used to store the value of each point. This map is protected by a mutex during update operations.

The definition of the Open Dyno application class is shown below. OpenDynoApp inherits from the the Poco::Application class.

class OpenDynoApp: public Application
{
public:
    OpenDynoApp();
    //Methods for creating and changing scripts 
    void        UpdateTestScript(RTScript* testscript);
    void        UpdateUserScript(RTScript* userscript);
    void        UpdateControlScript(RTScript* controlscript);
    RTScript*   CreateScriptEngine(string engineName, string fileName, std::vector<std::string>& neededFunctions);

    //Methods to change point and alarm objects at runtime
    void        UpdatePoints(Points* newPoints);
    void        UpdateAlarms(Alarms* newAlarms);

    //Data log management
    void        AddDataLog(std::string name, DataLogFile* logFile);
    bool        RemoveDataLog(std::string name);
    std::string GetDataLogNames();
    DATALOGMAP& OpenDynoApp::GetDataLogs();

protected:
    //Application management
    void        initialize(Application& self);
    void        uninitialize();
    void        reinitialize(Application& self);
    int         main(const std::vector<std::string>& args);
    
private:
    //Variable that holds the loop rate
    double             _loopRate;

    //Lock to protect access to global dataPoints
    //This is passed to the other threads 
    Poco::Mutex        _lock;
    //Lock to protect multiple access to the datalogs
    Poco::Mutex        _datalogLock;
    
    //Thread classes, all launched from the thread pool
    Poco::ThreadPool   _pool;
    ClientComms*       _clientcomms;
    MessageComms*      _messagecomms;
    DataLog*           _dataLog;
    Retentive*         _retentive;

    //Events to trigger other threads
    Poco::Event        _dataLogEvent;
    
    //UDP Socket to send messages to GUI
    ClientMessageSend* _clientMessage;
    
    //Point and Alarm objects and variables
    Points*            _myPoints;
    Alarms*            _myAlarms;
    int                AlarmCnt;
    int                noAlarmCnt;
    std::string        alarmNames;
    std::string        oldAlarms;
    
    //Script objects
    RTScript*          _testScript;
    RTScript*          _controlScript;
    RTScript*          _userScript;

    //Time metrics
    Poco::Stopwatch watch;
    Poco::Timestamp::TimeDiff startTime;
    Poco::Timestamp::TimeDiff prevStartTime;
    Poco::Timestamp::TimeDiff acqTimeDiff;
    Poco::Timestamp::TimeDiff pointTimeDiff;
    Poco::Timestamp::TimeDiff alarmTimeDiff;
    Poco::Timestamp::TimeDiff scriptTimeDiff;

    //Methods called during a scan of the RT Engine
    bool _helpRequested;
    void TimedCall(Timer& timer);
    void CreateGlobalDBPoints();
    void AlarmsUpdate();
    void CheckForFirstScan();
    void StepAndTimeUpdate();
    void ExecuteScripts();
      
    //Pointers to values in the global point database
    double* lastStep;
    double* currentStep;
    double* FirstScan;
    double* RUN;
    double* HOLD;
    double* IDLE;
    double* StepTime;
    double* TimeStep;
    double* StepLength;
    double* Step;
    double* NEXT_STEP;
    double* ElapsedTime;
    double* dbPointCnt;
    double* acqTime;
    double* PointUpdTime;
    double* AlarmUpdTime;
    double* ScriptUpdTime;
    double* TotalUpdTime;

    //Messages and datalogs
    std::map<std::string, OpenDynoMessage* > messages;
    std::map<std::string, DataLogFile* >     dataLogs;   
}; 

The pointers to values in the global points map or database are initialized during the construction of the OpenDynoApp class. This is shown below. This is done so that the look up in the std::map only occurs a single time. This provides a fairly large performance boost. The same optimization is done during the initialization of each point and alarm. The application never deletes points from the map so the pointer will be valid at all times.

    //Setup pointers to local variables
    //The points are created on first access
    lastStep      = &dataPoints["lastStep"];
    currentStep   = &dataPoints["currentStep"];
    FirstScan     = &dataPoints["FirstScan"];
    RUN           = &dataPoints["RUN"];
    HOLD          = &dataPoints["HOLD"];
    IDLE          = &dataPoints["IDLE"];
    StepTime      = &dataPoints["StepTime"];
    TimeStep      = &dataPoints["TimeStep"];
    StepLength    = &dataPoints["StepLength"];
    Step          = &dataPoints["Step"];
    NEXT_STEP     = &dataPoints["NEXT_STEP"];
    ElapsedTime   = &dataPoints["ElapsedTime"];
    dbPointCnt    = &dataPoints["dbPointCnt"];
    acqTime       = &dataPoints["acqTime"];
    PointUpdTime  = &dataPoints["PointUpdTime"];
    AlarmUpdTime  = &dataPoints["AlarmUpdTime"];
    ScriptUpdTime = &dataPoints["ScriptUpdTime"];
    TotalUpdTime  = &dataPoints["TotalUpdTime"]; 

The initialize method is shown below. It creates all of the point, alarm, and script objects as well as creating and starting the background threads. Notice the global dataPoints variable being used during the initialization of the points and alarms. The variable _pool is a thread pool that is used to launch all of the background threads.

 void OpenDynoApp::initialize(Application& self)
{
    //Poco application initialization
    //Load data from configuration file
    loadConfiguration(); 
    Application::initialize(self);

    //Setup internally used points in global database
    CreateGlobalDBPoints();

    //Create and start communication thread
    _clientcomms = new ClientComms("ClientComms", &_lock, *this);
    _pool.start(*_clientcomms, "ClientCommsThread");

    //The client message is for outgoing messages
    _clientMessage = new ClientMessageSend(); 
    
    //Create and initialize the Point objects
    _myPoints = new Points();
    _myPoints->Init(dataPoints);
    _myPoints->UpdatePoints(dataPoints);
    
    //Create and initialize the Alarm ojbects
    _myAlarms = new Alarms();
    _myAlarms->Init();
    _myAlarms->UpdatePoints(dataPoints);

    //Create the main control script
    //controlFunctions is a list of functions that must exist in the script
    std::vector<std::string> controlFunctions;
    controlFunctions.push_back("void Always()");
    controlFunctions.push_back("void AlarmGeneral()");
    controlFunctions.push_back("void AlarmCoast()");
    controlFunctions.push_back("void AlarmEmergency()");
    controlFunctions.push_back("void Idle()");
    _controlScript = CreateScriptEngine("control","control.as",controlFunctions);

    //Create the user script
    //userFunctions is a list of functions that must exist in the script
    std::vector<std::string> userFunctions;
    userFunctions.push_back("void User()");
    _userScript = CreateScriptEngine("user","user.as",userFunctions);

    //Create the test script
    //testFunctions is a list of functions that must exist in the script
    std::vector<std::string> testFunctions;
    testFunctions.push_back("void Step1()");
    _testScript = CreateScriptEngine("test","script.as",testFunctions);
  
    //Create and start I/O message communication thread
    _messagecomms = new MessageComms("MessageComms", &_lock, *this);
    _pool.start(*_messagecomms, "MessageCommsThread");

    //Create and start the retentive points thread
    //Load the retentive points last so that the previously stored values will be recalled
     cout << "Creating Retentive Point Thread." << endl;
    _retentive = new Retentive("Retentive", &_lock, *this);
    _pool.start(*_retentive, "RetentiveThread");
  
    //Create and start the data logging thread 
    _dataLog = new DataLog("DataLog", &_lock, &_datalogLock, *this, &_dataLogEvent);
    _pool.start(*_dataLog, "DataLogThread");
} 

The main control loop is a very simple block of code which is shown below. Most of the code shown is actually timekeeping information. Depending on the specific application there are a couple of ways to initiate this block of code. It could be started by a timer or it could be triggered from the arrival of an I/O data packet. This could be customized for each specific application. I've thought of having a configuration option which would allow the user to select the method but haven't got around to implementing that yet. In the simulator application the method is triggered by a timer set at 100Hz.

    {
    //Lock the mutex for exclusive access
        Poco::Mutex::ScopedLock locker(_lock);
        
    //Record timing information
        startTime     = watch.elapsed();
        acqTimeDiff   = (startTime - prevStartTime);
        prevStartTime = startTime;
        *acqTime      = (double)acqTimeDiff / 1000000.00;

    //Evaluate points
        _myPoints->UpdatePoints(dataPoints);
        pointTimeDiff = watch.elapsed();
        *PointUpdTime = (double)(pointTimeDiff - startTime) / 1000000.00;

    //Evaluate alarms
        AlarmsUpdate();
        alarmTimeDiff = watch.elapsed();
        *AlarmUpdTime = (double)(alarmTimeDiff - pointTimeDiff) / 1000000.00;

    //Run all scripts
        CheckForFirstScan();
        ExecuteScripts();
        StepAndTimeUpdate();
        scriptTimeDiff = watch.elapsed();
        *ScriptUpdTime = (double)(scriptTimeDiff - alarmTimeDiff) / 1000000.00;

    //Record timing information
        *dbPointCnt   = dataPoints.size();
        *ElapsedTime  = watch.elapsed();
        *TotalUpdTime = (double)(*ElapsedTime - startTime) / 1000000.00;        
        
        //Queue the real time data for the consumer thread to write to disk
        std::map<std::string, DataLogFile* >::iterator it;
        for (it= dataLogs.begin(); it != dataLogs.end(); it++)
        {
            (*it).second->QueueData();
        }
        //Post an event to the datalog thread to trigger logging
        _dataLogEvent.set();
    } 

The idle loop in the main thread looks like this. The big while loop could be customized to include any background processing or data collection if that is desired. Currently it just sleeps waiting for the ExitApplication point to change to a value of 1. The ExitApplication variable can be set either by remotely changing the value or pressing CTRL-C which is trapped by a signal handler.

 int OpenDynoApp::main(const std::vector<std::string>& args)
{
    //Setup application abort signal handlers
    #if POCO_OS   == POCO_OS_WINDOWS_NT
        SetConsoleCtrlHandler((PHANDLER_ROUTINE) consoleHandler, TRUE);
    #elif POCO_OS == POCO_OS_LINUX
        // Add signal interception
    #endif 

     cout << "***********************************************************" << endl;
     cout << "    ___                       ___                          " << endl;
     cout << "   /___\\_ __   ___ _ __      /   \\_   _ _ __   ___       " << endl;
     cout << "  //  // '_ \\ / _ \\ '_ \\    / /\\ / | | | '_ \\ / _ \\  " << endl;
     cout << " / \\_//| |_) |  __/ | | |  / /_//| |_| | | | | (_) |      " << endl;
     cout << " \\___/ | .__/ \\___|_| |_| /___,'  \\__, |_| |_|\\___/    " << endl;
     cout << "       |_|                        |___/                    " << endl;
     cout << "                    Version 1.0                            " << endl;
     cout << "             Copyright Tony Fountaine                      " << endl;
     cout << "                     2013-2014                             " << endl;
     cout << "                  www.opendyno.com                         " << endl;
     cout << "***********************************************************" << endl;
     cout << endl;
     cout << "Enter CTRL-C to shutdown application." <<endl;
     cout << endl;
   
    double* cycles = &dataPoints["Cycles"];
    double* exitApplication = &dataPoints["ExitApplication"];
    
    //This timer controls the application loop rate
    Timer timer(1000, 10);
    timer.start(TimerCallback<OpenDynoApp>(*this, &OpenDynoApp::TimedCall));
    prevStartTime = watch.elapsed();

    //Start watch to keep track of application time
    watch.start();

    //This is the main control loop that runs until application exit is requested.
    while(*exitApplication < 1.0)
    {
       //Wait for timer or semaphore
       //Get IO Data
       //Sleep for a little so that we don't consume endless CPU
       Poco::Thread::sleep(1000);
       //Add any housekeeping or watchdog code here
    } 

    //Stop the main loop timer
    timer.stop();
    return Application::EXIT_OK;
    //At this point Poco will call the unitialize method
} 

Before the application exits the un-initialize method is called. This is shown below where we see all the threads being stopped and objects deleted.

void OpenDynoApp::uninitialize()
{
    //Stop all background worker threads
    _clientcomms->cancel();
    _messagecomms->cancel();
    _dataLog->cancel();
    _retentive->cancel();
    _pool.stopAll();
    _pool.joinAll();

    //Delete all objects
    delete _clientMessage;
    delete _clientcomms;
    delete _messagecomms;
    delete _dataLog;
    delete _retentive;
    delete _userScript;
    delete _controlScript;
    delete _testScript;
    delete _myPoints;
    delete _myAlarms;

    //Poco default application uninitialize
    Application::uninitialize();
} 

That is the basic overview of the real time application. At startup all of the objects and threads are created and then the main timed or triggered loop executes for the remainder of the application. Finally all of the threads are stopped and all created objects are destroyed at shutdown.

Client Communication

The client communication thread (ClientComms class) allows a remote computer to send and receive data with the Open Dyno Application. There is no requirement for the computer to be remote. The commands could be sent from another application on the same computer. This is what the simulator application does. All of the commands are sent using an Ethernet UDP datagram. Some of the commands that are available are:

Get Data Points -> This sends all of the points and values in dataPoints in an ASCII format

Update Point Value -> Updates the value of a single point in dataPoints

Reload Test Script -> Update the test script.as

Reload User Script -> Reload the user.as script

Reload Points -> Reload all of the point definitions

Reload Alarms -> Reload all of the alarm definitions

Create Datalog -> Create a new data log

Remove Datalog -> Remove a data log

Get Datalogs -> Returns all of the active data logs

Each of the commands is a simple ASCII command that could be built and sent from any language. Commands are identified by an integer value. A command is simply the integer value followed by the parameters separated by a pipe character "|". For example the update variable command is integer 3 and takes two parameters, the point name and the new value. The full syntax for the command is "3|VariableName|NewValue". Open Dyno includes the Configuration Manager as well as the Visualization and Screen applications that are able to send these commands. Other commands are available and new ones are easy to create. Multiple commands can be sent by separating them on new lines.

There was a brief discussion about the global points std::map when discussing the application architecture above. The map serves as common location to store all of the data values for each point, alarm and script object to access. Having the values stored outside of the objects allows each object to be dynamically modified or updated without affecting the location of the data value in memory. This is important since we use pointers to optimize the speed of finding data. It also allows all of the objects to be dynamically modified at runtime. For example, since the points are all stored in a points collection object, a new points collection could be created in a background thread at runtime to replace the existing one. This is one of the functions of the communication thread. It creates the new objects in the background. If the object is created successfully it will lock the global mutex and then switch the point collection to the new object. This allows each object to be swapped out in a single cycle. This is very important since points, alarms and tests all need to be dynamically updated at runtime.

Message Communication

The message communication thread (MessageComms class) was created to add a high speed binary communication protocol for transferring large numbers of point values into and out of the real time application. Since the client communication protocol discussed earlier is based on ASCII commands it is much slower to process commands. The binary protocol for message communication uses an Open Dyno Message. These messages are predefined in the Configuration Manager application using the dialog below. The messages can have a direction of incoming or outgoing. If the message is outgoing it requires an update rate and a destination UDP port and IP address. All messages are identified by an ID and a Name. The name was only included to make it easier to diagnose any communication issues using WireShark.

The structure of the Open Dyno Message is very simple. It has a header followed by double values packed into a char array. The header structure is shown below. The MessageID and MessageName correspond to the dialog fields ID and Name above. The PointCount and MessageSize are used to validate the message before trying to parse it. The MessageCounter is an incrementing counter and can be used to detect missing messages. The Future variable is simply padding.

 struct UDPMessageHeader
{
    int  MessageID;              //Numerical message identifier
    int  MessageSize;            //Total number of bytes in message
    char MessageName[32];        //Text based message identifier
    int  PointCount;             //Count of the number of points in message
    int  MessageCounter;         //Incrementing counter
    char Future[32];             //32 bytes reserved for future use
}; 

The actual OpenDynoMessage class holds a vector of pointers to each of the points in the dataPoints map. This makes it very fast to build or parse a message when needed. There is a pre-allocated buffer for the message so no time is wasted allocating memory at runtime.

Data logging

It's difficult to find a simple data format to efficiently stream data to disk. The decision was made to create a very simple binary file for Open Dyno. The file is called Open Dyno Binary File and has an extension of "odbf" for lack of a better name. It contains a simple header followed by any number of Open Dyno Messages. This is the same Open Dyno Message used for communication. This allowed me to commonize both the communication and data logging functionality. There is an overhead of 80 bytes of header for each sample. It is possible to have multiple simultaneous datalogs. I have ran with 10 logs collecting all of the points at 100Hz without any issue.

The header of the file contains the following information.

Buffer Size -> 4 byte integer -> Size in bytes of the Open Dyno Message

Header Size -> 4 byte integer -> Size in bytes of the file header

Channel Count -> 4 byte integer -> Number of points in each Open Dyno Message

Sample Rate -> 4 byte integer -> Rate the data was recorded (1, 10, or 100Hz)

Point Names -> variable length string -> Names of points separate by line feeds

I created the SDF Viewer application that is able to view these files as well as many other file formats including MDF, TDMS, CSV, and DAT.

Logging data in the main control loop can cause a large amount of jitter and may even exceed the sample time. It can take an undetermined amount of time for the disk write to take place. To alleviate this issue a producer consumer pattern was used. Each data log makes use of a DataLogFile class. When these classes are created there is a buffer of 100 OpenDynoMessages that are pre-allocated. This reduces any need for memory allocation in the main loop. Each of the OpenDynoMessage objects is transferred into and out of a queue. The main thread requests a message from the buffer and then inserts it into the queue. The data log thread takes the messages out of queue and copies the data to a local buffer followed by a write to disk. The local copy minimizes the time the queue mutex needs to be locked.

The main thread calls the QueueData method of the DataLogFile class.

void DataLogFile::QueueData()
{    
    if (!_ptr_myfile)
    {
        return;
    }
    bool writeData = false;
    if(_decimation > 0)
    {
        if(_writeRequests % _decimation == 0) writeData = true;
    }
    else
    {
        writeData = true;
    }
    _writeRequests++;

    if(writeData && _message != NULL)
    {
        char* message = _message->GetMessage();
        {
            //Scope the mutex lock to minimize lock time 
            Poco::FastMutex::ScopedLock locker(_mutexQueue);
            _messageQueue.push(message);
        }    
    }
} 

The QueueData method above in turn calls the GetMessage method of the OpenDynoMessage ( _message->GetMessage();)

char* OpenDynoMessage::GetMessage()
{
    if(bufferPool != NULL)
    {
        try
        {
            if(_buffered) buffer = (char*)bufferPool->get();
            //Write the current time into future
            time_t rawtime;
            time( &rawtime);    
            sprintf(header.Future, "%s", ctime(&rawtime));
            //Increment message counter
            header.MessageCounter++;
            //Copy the header into the buffer
            memcpy(&buffer[0],&header,sizeof(UDPMessageHeader));
            //Update all of the data points
            double* valueArray = (double*)&buffer[HeaderByteSize];
            for(unsigned int i = 0; i < ptrPoints.size(); i++)
            {
                valueArray[i] = *ptrPoints[i];
            }
        }
        catch(exception& e)
        {
            std::cout << e.what() << std::endl;
        }
    }
    return buffer;
} 

The data log thread is triggered by an event from the main loop which then calls the DataLogFile::WriteData method below

 void DataLogFile::WriteData()
{  
    if (!_ptr_myfile)
    {
        return;
    }

    char* message;
    int more = 1;
    while(more>0)
    {
        {
            //Scope the mutex to minimize lock time 
            Poco::FastMutex::ScopedLock locker(_mutexQueue);
            more = _messageQueue.size(); 
            if(more>0 && _message != NULL)
            {
                message = _messageQueue.front();
                //Copy the message into a temporary buffer so we don't lock on a file write
                memcpy(_buffer, message, _message->BufferSize());
                _messageQueue.pop();
                _message->ReleaseMessage(message);
            }
        }

        if(more>0 && _message != NULL)
        {
            fwrite(_buffer, _message->BufferSize(), 1, _ptr_myfile);
            _writes++;  
        }

        //Check if we need to create a new log file
        if(_writes > _maxWrites)
        {
            cout << "\nAutomatically creating new datalog " << _logName << " after "  << _writes << " points." << std::endl;
            CreateLogFile();
            _writes = 0;
            _writeRequests = 0;
        }
    }   
} 

The WriteData method about calls the ReleaseMessage method to return the message back to the buffer pool for reuse.

void OpenDynoMessage::ReleaseMessage(char* message)
{
    if(_buffered && bufferPool != NULL)
    {
        bufferPool->release(message);
    }
} 

The communication thread use the following methods of the OpenDynoApp class to manage the creation an deletion of DataLogFile objects. Notice the mutex locks to ensure exclusive access to the data logs.

void OpenDynoApp::AddDataLog(std::string name, DataLogFile* logFile)
{
    Poco::Mutex::ScopedLock mainlocker(_lock);
    Poco::Mutex::ScopedLock dataloglocker(_datalogLock);
    //Remove any existing datalog and delete it
    std::map<std::string, DataLogFile* >::iterator it;
    it = dataLogs.find(name);
    if(it != dataLogs.end())
    {
        dataLogs.erase(it);
        delete (*it).second;
    }

    //Add the new data log
    dataLogs[name] = logFile;
}

bool  OpenDynoApp::RemoveDataLog(std::string name)
{
    Poco::Mutex::ScopedLock mainlocker(_lock);
    Poco::Mutex::ScopedLock dataloglocker(_datalogLock);
    bool result = false;
    //Remove any existing datalog and delete it
    std::map<std::string, DataLogFile* >::iterator it;
    it = dataLogs.find(name);
    if(it != dataLogs.end())
    {
        dataLogs.erase(it);
        delete (*it).second;
        result = true;
    }
    return result;
} 

Data Points and Alarms

Since there are many similarities between points and alarms, only points will be discussed. Data points are all based on an abstract class called Conversion whose definition is shown below.

class Conversion 
{
public:
    Conversion();
    virtual ~Conversion();
    virtual void                Convert(DATAPOINTS &db)    = 0;
    virtual void                Initialize(DATAPOINTS &db) = 0;
} 

There are four derived classes named Linear, Interpolation, Formula, and ScratchPad. The definition of the Linear class is shown below.

 class Linear : public Conversion  
{
public:
    Linear();
    virtual ~Linear();
    virtual void Convert(DATAPOINTS &db);
    virtual void Initialize(DATAPOINTS &db);
    
    double      m;
    double      b;
    double      max;
    double      min;
    std::string PointName;
    std::string PointNameAvg;
    std::string RawName;
    double*     point;
    double*     avgPoint;
    double*     rawPoint;
}; 

The implementation of Linear is shown below. All of the point classes use pointers to values in the std::map of points to access other point values or to update their own value. The pointers are referenced in the Initialize method shown below. This eliminates the time to find the point in the map when executing the Convert method.

 Linear::Linear()
{
    point    = 0;
    avgPoint = 0;
    rawPoint = 0;
}

Linear::~Linear()
{

}

void Linear::Initialize(DATAPOINTS &db)
{
    if(point == 0)
    {
        point    = &db[PointName];
        rawPoint = &db[RawName];
        if(avgPoints > 0)
        {
            avgPoint = &db[PointNameAvg];
        }
    }
}

 void Linear::Convert(DATAPOINTS &db)
{
    double temp = 0.0;
    double sum  = 0.0;

    if(point == 0)
    {
        Initialize(db);
    }

    temp = m * *rawPoint +b;
    if(temp > max)      {*point = max;}
    else if(temp < min) {*point = min;}
    else                {*point = temp;}
    
    if(isAvg == 1)
    {
        *point = filter(*point );        
    }

    if(avgPoints > 0)
    {
        *avgPoint = average(*point);    
    }
} 

All of the point types are stored in a point collection called Points. This was seen previously in the discussion of the main loop. The initialization of the points uses all of the information entered in the GUI and stored in the points table of the database. This is shown below. All of the SQLite database calls make use of the CppSQLite project on CodeProject. The points are selected from the database in order of priority to ensure they are calculated in the proper order. Many points will reference other points in their computation that need to be calculated first.

 void Points::Init(DATAPOINTS &db)
{
    CppSQLite3DB sqliteDB;
    int ConversionType;

    std::string dfFileName = Path::current() + "katerina.db3";
    const char *gszFile = dfFileName.c_str();
            
    // Clean up all of the old points 
    std::vector<Conversion*>::iterator it;
    for (it=points.begin(); it < points.end(); it++)
    {
        delete *it;
        points.erase(it);
    }
    points.clear();
            
    sqliteDB.open(gszFile);
   
    CppSQLite3Table t = sqliteDB.getTable("SELECT Units," 
                                                 "Name," 
                                                 "Conversion," 
                                                 "Description," 
                                                 "Minimum," 
                                                 "Maximum," 
                                                 "m," 
                                                 "PointLinear," 
                                                 "b," 
                                                 "PointInterp," 
                                                 "DataTable," 
                                                 "Formula," 
                                                 "Average," 
                                                 "Priority," 
                                                 "PtsToAvg " 
                                                 " FROM points ORDER BY priority");

    std::cout << "\nDatabase point count = " << t.numRows() << std::endl;

    for (int row = 0; row < t.numRows(); row++)
    {
        t.setRow(row);
        ConversionType = t.getIntField(2);
        //Pre-create point in database 
        double temp = db[t.getStringField(POINTNAME)];
        //std::cout << "\nCreated point " << t.getStringField(POINTNAME) << std::endl;
        switch(ConversionType)
        {    
        case LINEAR:
            {
                Linear*    myLinear;
                myLinear = new Linear;
                myLinear->m            = t.getFloatField(MFIELD);
                myLinear->b            = t.getFloatField(BFIELD);
                myLinear->min          = t.getFloatField(MINFIELD);
                myLinear->max          = t.getFloatField(MAXFIELD);
                myLinear->PointName    = t.getStringField(POINTNAME);
                myLinear->PointNameAvg = myLinear->PointName + "_Avg";
                myLinear->RawName      = t.getStringField(POINTLINEAR);
                
                if( myLinear->RawName == "")
                {
                    std::cout << "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" << std::endl;
                    std::cout << myLinear->PointName 
                              << " is missing Input Point   " << std::endl;
                    std::cout << "!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!!" << std::endl;    
                }
                    
                myLinear->isAvg     = t.getIntField(AVERAGE);
                myLinear->avgPoints = t.getIntField(PTSTOAVG);
                points.push_back(myLinear);
                break;
            }
        case INTERP:
            {
                std::string Query = "SELECT Raw, EUUnits FROM "; 
                Query +=  t.getStringField(DATATABLE);
                Query += " ORDER BY Raw";
                CppSQLite3Table tD = sqliteDB.getTable(Query.c_str());
                Interpolate* myInter;
                DoublePair*  myCCPair;
                myInter = new Interpolate;
                myInter->PointName    = t.getStringField(POINTNAME);
                myInter->PointNameAvg = myInter->PointName + "_Avg";
                myInter->RawName      = t.getStringField(POINTINTERP);
                myInter->min          = t.getFloatField(MINFIELD);
                myInter->max          = t.getFloatField(MAXFIELD);
                myInter->isAvg        = t.getIntField(AVERAGE);
                myInter->avgPoints    = t.getIntField(PTSTOAVG);
                                         
                for(int i=0;i< tD.numRows();i++)
                {    
                    tD.setRow(i);
                    myCCPair = new DoublePair;
                    myCCPair->Raw     = tD.getFloatField(0);
                    myCCPair->EUUnits = tD.getFloatField(1);
                    myInter->table.push_back(myCCPair);          
                }

                points.push_back(myInter);
                break;
            }
            
        case FORMULA:
            {   
                Formula*    myFormula;
                myFormula = new Formula;
                myFormula->PointFormula = t.getStringField(FORMUL);
                myFormula->PointName    = t.getStringField(POINTNAME);
                
                myFormula->PointNameAvg = myFormula->PointName + "_Avg";
                myFormula->min          = t.getFloatField(MINFIELD);
                myFormula->max          = t.getFloatField(MAXFIELD);
                myFormula->isAvg        = t.getIntField(AVERAGE);
                myFormula->avgPoints    = t.getIntField(PTSTOAVG);
                myFormula->Init();
                points.push_back(myFormula);
                break;                        
            }
                
            case SCRATCHPAD:
            { 
                ScratchPad*    myScratchPad;
                myScratchPad = new ScratchPad;
                myScratchPad->PointName    = t.getStringField(POINTNAME);
                myScratchPad->PointNameAvg = myScratchPad->PointName + "_Avg";
                myScratchPad->min          = t.getFloatField(MINFIELD);
                myScratchPad->max          = t.getFloatField(MAXFIELD);
                myScratchPad->avgPoints    = t.getIntField(PTSTOAVG);
                points.push_back(myScratchPad);
                break;                                        
            }
        }
    }           
    sqliteDB.close();
} 

The main loop calls the UpdatePoints method of the Points class which is shown below. This basically iterates over each of the points and calls its Convert method.

 void Points::UpdatePoints( DATAPOINTS &db)
{
    std::vector<Conversion*>::iterator it;
    for (it = points.begin(); it < points.end(); it++)
    {
        (*it)->Convert(db);
    }
} 

Since all of the point types are based on the Conversion class the application can be extended to include other points of arbitrary complexity. I have considered adding PID type points in the past but currently just use scripts for this functionality. The advantage of using scripts is that the code can be easily updated if needed. The disadvantage is that the parameters for the PID controller need to be defined individually. If a PID type point was created it would allow the points to be defined together.

The most interesting point type is the Formula point. This point makes use of the muParser mathematical engine to allow calculate equations. The equations can include other points in their definition. This was discussed in the previous article. The implementation for the Formula point is shown below. It's surprisingly small for the amount of functionality that it provides. The one line of code that does the actual calculation is parser.Eval(); This simplicity is all due to the functionality provided by muParser.

 extern double* AddVariable(const char_type *a_szName, void *a_pUserData);

Formula::Formula()
{
    AlarmStored = 0;
    point       = 0;
    avgPoint    = 0;
    rawPoint    = 0;
}

Formula::~Formula()
{
}

void Formula::Init()
{
    error = 0;
    try
    {
        parser.SetVarFactory(AddVariable, this);
        parser.SetExpr(PointFormula);
        parser.Eval();
    }
    catch(Parser::exception_type &e)
    {
        error = 1;
        std::cout << "Equation Parser error for point or alarm: " 
                  << PointName << e.GetMsg() << std::endl;
    }
}


void Formula::Initialize(DATAPOINTS &db)
{
    if(point == 0)
    {
        point = &db[PointName];
        if(avgPoints > 0)
        {
            avgPoint = &db[PointNameAvg];
        }
    }
}

 void Formula::Convert(DATAPOINTS &db)
{
    if(point == 0)
    {
        Initialize(db);
    }

    try
    {
        double sum = 0.0;
        
        if(error == 0) {parser.Eval();}
        
        if(*point > max)      {*point = max;}
        else if(*point < min) {*point = min;}
               
        if(isAvg == 1)
        {
            *point = filter(*point);           
        }

        if(avgPoints > 0)
        {
            *avgPoint = average(*point);           
        }
    }
    catch(Parser::exception_type &e)
    {
        std::cout  << "Equation Parser error for point or alarm: " 
                   << PointName << "   " << e.GetMsg() << std::endl;
    }
} 

In the code above parser is a muParser object. In the Init method the parsers SetVarFactory method is called that allows it to find points in the global points map. This method is shown below. The function always returns a pointer even if the point didn't exist. There is an error message that indicates the issue. The Configuration manager performs a validation on equation points to ensure that all of the points exist so this shouldn't ever happen.

 //---------------------------------------------------------------------------
// Factory function for creating new parser variables
double *AddVariable(const char_type *a_szName, void *a_pUserData)
{
    Formula* temp = (Formula*) a_pUserData;
    DATAPOINTS::iterator it;                            
    it = dataPoints.find(a_szName);

    if (it == dataPoints.end())
    {    
        std::cout << "Point - " << temp->PointName 
                  << "\n\tmuParser requested an undefined point " 
                  << a_szName << std::endl;
    }

    return(&dataPoints[a_szName]);
}

Scripting

Open Dyno makes use of the AngelScript scripting language to allow it to perform arbitrarily complex logic. AngelScript is a statically typed language that is very similar to C++. AngelScript compiles script text to byte code prior to execution. When discussing the main loop, the code that creates each RTScript object was shown. This is repeated again here for the user script.

//Create the user script
//userFunctions is a list of functions that must exist in the script
std::vector<std::string> userFunctions;
userFunctions.push_back("void User()");  
_userScript = CreateScriptEngine("user","user.as",userFunctions);  

The implementation of the CreateScriptEngine method is shown below.

RTScript* OpenDynoApp::CreateScriptEngine(string engineName, 
                                          string fileName, 
                                          std::vector<std::string>& neededFunctions)
{
    RTScript* tempScript = NULL;

    std::cout << "Starting build of " + engineName + " script engine." << std::endl;
    std::string scriptFileName = Path::current() + fileName;
    tempScript = new RTScript(scriptFileName);
    std::cout << "\tAdding variables to " + engineName + " script engine." << std::endl;
    tempScript->AddVariables(dataPoints);
    std::cout << "\tLoading script into " + engineName + " script engine." << std::endl;

    int result = tempScript->ConfigureEngine(neededFunctions);
    if(result <= 0)
    {
        std::cout << "\tERROR building " + fileName + "\n" << std::endl;
        tempScript->Release();
        delete tempScript;
        tempScript = NULL;
    }
    else
    {
        std::cout << "\tDone " + engineName + " script build.\n" << std::endl;
    }
    return tempScript;
} 

Each RTScript object contains its own AngelScript engine. The constructor of the RTScript class takes the name of the script file and is shown below. The constructor creates the script engine and registers any required user functions with the functions RegisterXXX. It also sets a call back function that is used to print any error messages from the script. The ClientMessageSend class is used to allow the script engine to send UDP messages directly to the client.

RTScript::RTScript(string fileName):_fileName(fileName) {
    // Initialize member variables
    engine = 0;
    ctx    = 0;
    _clientMessage = new ClientMessageSend(); 
    engine = asCreateScriptEngine(ANGELSCRIPT_VERSION);
    if( engine == 0 )
    {
        std::cout << "\tFAILED to create script engine ." << std::endl;
        return;
    }

    // The script compiler will write any compiler messages to the callback.
    engine->SetMessageCallback(asFUNCTION(MessageCallback), 0, asCALL_CDECL);

    RegisterStdString(engine);
    RegisterScriptMath( engine);
    RegisterScriptArray(engine,1);
    RegisterClientMessage(engine);
    RegisterOpenDyno(engine);
}

After the script engine is created and all of the functions have been registered, the next step is to link all of the points to the engine so that they are accessible by name in the script. This is done with the AddVariables method below. This iterates over all the points and registers them as global variables of type double in the script engine using the RegisterGlobalProperty method of the engine.

void RTScript::AddVariables(DATAPOINTS &db)
{
        // Configure the script engine with all the functions,
        // and variables that the script should be able to use.
        std::string myPoint;
        DATAPOINTS::iterator it;
        engine->RegisterGlobalProperty("int intStepTime", &intStepTime);
        for (it=db.begin(); it != db.end(); it++)
        {
            myPoint.erase();
            myPoint = "double  ";
            myPoint += it->first;
            engine->RegisterGlobalProperty(myPoint.c_str(), &db[ it->first]) ;
        }
} 

The final step in creating the script engine is to actually open the file and compile it. This is done below as part of he ConfigureEngine method. This method also creates a context for executing script functions and caches pointers to each script function in a vector named myFunctions. This makes it much faster to call script functions at runtime.

int RTScript::ConfigureEngine(std::vector<std::string>& requiredFunctions)
{
    int funcCount=0;

    // Compile the script code
    r = CompileScript();
    if( r < 0 )
    {
        return r;
    }

    ctx = engine->CreateContext();
    if( ctx == 0 )
    {
        std::cout << "\tFAILED to create the context." << std::endl;
        return -1;
    }

    funcCount = engine->GetModule(0)->GetFunctionCount();
    std::cout << "\tThe script contains " << funcCount << " global functions." <<  std::endl;
    for(int theCount =0 ; theCount < funcCount; theCount++)
    {
        asIScriptFunction*  theFuncId = engine->GetModule(0)->GetFunctionByIndex(theCount);
        myFunctions[theFuncId->GetName()] =  theFuncId;
    }

    //Make sure all of the required functions exist in the script
    return CheckFunctions(requiredFunctions);
    //return 1;
} 

The actual compile function is shown below. It makes use of the AngelScript add on class CScriptBuilder. The script builder is a nice utility class. It adds the ability to use precompiler directives such as #include as well as meta data. This makes it much easier to separate control logic into smaller modules and classes.

int RTScript::CompileScript()
{
    int r;

    // The builder is a helper class that will load the script file, 
    // search for #include directives, and load any included files as 
    // well.
    CScriptBuilder builder;

    try
    {
        // Build the script. If there are any compiler messages they will
        // be written to the message stream that we set right after creating the 
        // script engine. If there are no errors, and no warnings, nothing will
        // be written to the stream.
        r = builder.StartNewModule(engine, 0);
        if( r < 0 )
        {
            cout << "\tFAILED to start new module" << endl;
            return r;
        }
        r = builder.AddSectionFromFile(_fileName.c_str());
        if( r < 0 )
        {
            cout << "\tFAILED to add script file" << endl;
            return r;
        }
        r = builder.BuildModule();
        if( r < 0 )
        {
            cout << "\tFAILED to build the module" << endl;
            return r;
        }
    }
    catch(...)
    {
        std::cout << "\t!!!! An unknown error occurred while building the test scripts." << std::endl;
    }
    return 0;
} 

The main loop of the application calls functions in the script using the RunFunction method below. This function isn't directly called. Other Run functions exist that take the function name and look it up in a map to find the pointer to the actual script function which is then passed to the RunFunction method.

void RTScript::RunFunction(asIScriptFunction *fID)
{
    // Prepare the script context with the function we wish to execute. Prepare()
    // must be called on the context before each new script function that will be
    // executed. Note, that if you intend to execute the same function several
    // times, it might be a good idea to store the function id returned by
    // GetFunctionIDByDecl(), so that this relatively slow call can be skipped.

    r = ctx->Prepare(fID);
    if( r < 0 )
    {
        std::cout << "\tFAILED to prepare the context." << std::endl;
        ctx->Release();
        engine->Release();
        ctx    = 0;
        engine = 0;
        return;
    }

    r = ctx->Execute();

    if( r != asEXECUTION_FINISHED )
    {
        // The execution didn't finish as we had planned. Determine why.
        if( r == asEXECUTION_ABORTED )
            std::cout << "\tThe script was aborted before it could finish. Probably it timed out." << std::endl;
        else if( r == asEXECUTION_EXCEPTION )
        {
            std::cout << "\tThe script ended with an exception." << std::endl;

            // Write some information about the script exception
            asIScriptFunction *func = ctx->GetExceptionFunction();
            cout << "\tfunc: " << func->GetDeclaration() << endl;
            cout << "\tmodl: " << func->GetModuleName() << endl;
            cout << "\tsect: " << func->GetScriptSectionName() << endl;
            cout << "\tline: " << ctx->GetExceptionLineNumber() << endl;
            cout << "\tdesc: " << ctx->GetExceptionString() << endl;
        }
        else
        {
            std::cout << "\tThe script ended for some unforeseen reason (" 
                      << r << ")." << std::endl;
        }
    }
} 

Points of Interest

There are a lot of small details that haven't been covered in the article. The main scope of how the real time application executes was described. The link was made between the data entered in the GUI and how it used by the real time application. The next article will show how the HMI applications Open Dyno Visualize and Screen are used to communicate over Ethernet with the real time application. That should provide enough detail for anyone to either use or extend Open Dyno for their own purpose.

The dynamic nature of the real time application was also shown. This is one of the best features of Open Dyno. In order to run an engine test cell or some other kind of test it is important to be able to reconfigure the system without shutting down the application. Shutting down and starting an engine to make a change can result a lot of lost testing time while the engine warms back up. This dynamic functionality is also used to adjust alarms and modify points at runtime.

Both muParser and AngelScript are high performance libraries. The syntax of muParser is the same as basic algebra so end users don't need to learn any complex new syntax to describe simple equations. AngelScript syntax is nearly identical to C++. Help can be found all over the Internet for C++. The added advantage of using AngelScript is that any script code could be compiled directly in the core if the performance is not good enough. I haven't had to do this yet. As a mater of fact there reverse has been the case. I keep most of the algorithms and logic in script since it is more than fast enough. It makes it much easier to modify on a moments notice.

History

April 18, 2014 -> Initial release.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Anthony Fountaine
Engineer
Canada Canada
Tony is a research and development engineer at the Powertrain Engineering Research and Development Center (PERDC) in Windsor, Ontario.

Comments and Discussions

 
QuestionSource for the OpenDynoVisualize PinmemberThomas Hjortby21-Apr-14 2:47 
AnswerRe: Source for the OpenDynoVisualize PinmemberAnthony Fountaine21-Apr-14 3:18 
AnswerRe: Source for the OpenDynoVisualize PinmemberAnthony Fountaine22-Apr-14 3:07 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

| Advertise | Privacy | Mobile
Web02 | 2.8.140821.2 | Last Updated 19 Apr 2014
Article Copyright 2014 by Anthony Fountaine
Everything else Copyright © CodeProject, 1999-2014
Terms of Service
Layout: fixed | fluid