How To Add Simple Web-enabled 2D/3D Dashboards to your Natively Built Application

• Download source code - 2.96 MB

Introduction

Recently, after seeing a lot of advertisement for web-based visualization dashboards, I set out to answer the following question. Is there an easy way to provide simple web-enabled 2D/3D visualization dashboards for natively built C/C++ applications for free?

Background

I have reviewed many web-based visualization solutions (dashboard software). Many of them require supporting client side technologies such as .NET, Flash, Java, etc... In our particular situation, we did not want to burden our end-users with the responsibility of maintaining client-side frameworks. Another issue was the availability of these technologies for all of the various browser implementations and their supported platforms. In addition, many of the solutions rendered data on the client side, an approach that suffers in performance for huge data sets. When I talk of huge data sets, I am talking about tens of thousands of data points. With no budget for development, the biggest factor was cost.

So, how do you provide a web-enabled 2D-3D data visualization solution for natively built applications on a zero budget? One answer, write your own embeddable web-server that can take 2D or 3D data, render it to an image, and send the rendered image to a client-browser for visualization. This might sound like a major development effort, but it's not. In fact, the solution presented here took less than 15 minutes to develop.

The Solution

The diagram above illustrates our approach. In our approach, we use three components:

• gnuplot
• an embedded web-server
• bubble (plug-in)

Let's begin with gnuplot. gnuplot is a powerful natively built plotting tool that has been around for years, runs on most platforms, and is well documented (there are books on gnuplot). It has an extensive scripting language that is conducive to dynamic plots, a requirement for even the most basic dashboard solution. In addition, it is free to distribute -- no royalties.

Since we are talking about adding the functionality to natively built applications, the simplest approach is an embedded one. We use the Snorkel SDK to develop the embedded server component. Snorkel (http://snorkelembedded.webs.com/) is an embedded/application server-SDK that contains a powerful set of APIs designed to make coding solutions like this one easy, and as with gnuplot, it is also free.

Data visualization is a useful tool that we will likely want to reuse in future projects. For the purpose of reusability, we write the renderer as a plug-in, that is, a bubble in Snorkel terminology. The bubble will act as an interface between our embedded web-server and gnuplot.

For those who are not familiar with the term bubble, bubbles are shared objects that encapsulate and associate one or more functions with file extensions (MIME types), URIs, and/or proprietary protocols. Not to be confused with CGIs, bubbles run within the web-server not as a separate process.

In general, bubbles contain a single exported function, bubble_main. As part of the load process, the Snorkel runtime calls each bubble's bubble_main routine to perform any necessary initialization and to identify the functional purpose of the bubble. It is from this routine that associations between MIMEs, URIs, and user defined protocols are established.

In this project, we associate the extension gpng with gnuplot script files. A gnuplot script is a text file that contains a set of instructions that gnuplot executes to produce an image. We register the extension-MIME type in the bubble's bubble_main routine using the API function snorkel_obj_set.

  .     .
 51    #define GNUENV          "GNUPLOT"
 52    #define TERMINAL        "set terminal png transparent truecolor enhanced"
 53    #define GNU_ARGS        " -e "
 54    #define EXTENSION       "gpng"
 55    #define TYPE            FILE_PNG
 56    #define MODE            "wt"
  .
  .
241
242
243    byte_t SNORKEL_EXPORT
244    bubble_main (snorkel_obj_t server)
245    {
246      snorkel_obj_set (server,
247                       snorkel_attrib_mime,
248                       EXTENSION,
249                       TYPE, encodingtype_binary, gnuplot_gpng);
250      return 1;
251    }

In the call to snorkel_obj_set, we identify the extension, the file type, the encoding, and the routine responsible for rendering MIMEs of type gpng. Basically, the call instructs the server to render all files whose filenames have the extension .gpng with the function gnuplot_gpng.

The gnuplot_gpng function takes incoming requests for URIs of type gpng and sends their URLs to gnuplot for rendering/execution. gnuplot converts incoming gpng URLs to temporary PNG (Portable Network Graphics) files, and the function gnuplot_gpng streams them back to the requesting client's browser before deleting them.

 51    #define GNUENV          "GNUPLOT"
 52    #define TERMINAL        "set terminal png transparent truecolor enhanced"
 53    #define GNU_ARGS        " -e "
 54    #define EXTENSION       "gpng"
 55    #define TYPE            FILE_PNG
 56    #define MODE            "wt"
 57
 58    #if !defined(WIN32) && !defined(WIN64)
 59    #define _popen popen
 60    #define _pclose pclose
 61    #define _dir_sep '/'
 62    #define sprintf_s snprintf
 63    #else
 64    #define _dir_sep '\\'
 65    #endif
  .
  .
  .
117
118    /**
119     *
120     * gnuplot_gpng
121     * called by server to render gpng (gnuplot script) files as
122     * png
123     *
124     **/
125    call_status_t
126    gnuplot_gpng (snorkel_obj_t http,
127                  snorkel_obj_t connection, char *pszurl)
128    {
129      unsigned int canary[] = _CANARY_VALUE;
130      char szwork_dir[512];
131      char sztmpname[256];
132      char szcommand[1024];
133      char *gnuplot = 0;
134      char *psz     = 0;
135      FILE *fd      = 0;
136      FILE *gnupipe = 0;
137      char *psztmp  = 0;
138      char *pszfile = 0;
139
140
141      /*
142       * set our working directory, the parent
143       * directory of the URL
144       *
145       */
146      sztmpname[0]  = 0;
147      szcommand[0]  = 0;
148      szwork_dir[0] = 0;
149      strncat (szwork_dir, pszurl, sizeof (szwork_dir));
150      pszfile = strrchr (szwork_dir, _dir_sep);
151      if (pszfile)
152      {
153        *pszfile = 0;
154     pszfile++;
155      }
156      else
157      {
158       _stkchk;
159       return ERROR_STRING ("could not determine working directory\r\n");
160      }
161
162
163      /*
164       * get the location of gnuplot
165       */
166      gnuplot = get_pgnuplot ();
167      if (!gnuplot)
168        {
169          _stkchk;
170          return ERROR_STRING ("GNUPLOT not set\r\n");
171        }
172
173      /*
174       *
175       * create a temporary file to receive
176       * our png data
177       *
178       */
179      tmpnam (sztmpname);
180
181      psztmp =
182        (sztmpname[0] == _dir_sep) ? &sztmpname[1] : sztmpname;
183
184      /*
185       *
186       * crank out the command line
187       *
188       */
189      sprintf_s (szcommand, sizeof (szcommand),
190              "%s\"cd '%s';%s;set output '%s';load '%s'\"",
191        gnuplot, szwork_dir, TERMINAL, psztmp, pszfile);
192
193    #if defined(_DEBUG) || defined(DEBUG)
194      printf("%s\n",szcommand);
195    #endif
196
197      /*
198       *
199       * run our gnuplot session
200       *
201       */
202      gnupipe = _popen (szcommand, MODE);
203      if (!gnupipe)
204        {
205          _stkchk;
206          return
207            ERROR_STRING
208            ("error: could not open gnuplot pipe\r\n");
209        }
210      fflush (gnupipe);
211      _pclose (gnupipe);
212
213
214      /*
215       *
216       * all done, stream the plot to
217       * the browser
218       *
219       */
220      sprintf_s (szcommand, sizeof (szcommand), "%s%s", szwork_dir,
221                 sztmpname);
222
223
224      if (snorkel_file_stream (connection,
225                               szcommand,
226                               0,
227                               SNORKEL_BINARY) == SNORKEL_ERROR)
228        {
229          remove (szcommand); /* remove the temporary file */
230          _stkchk;
231          return ERROR_STRING ("could not stream data\r\n");
232        }
233
234      /* this is it, we are done... it doesn't get much easier
235         than that */
236      remove (szcommand); /* remove the temporary file */
237      _stkchk;
238
239      return HTTP_SUCCESS;
240    }

Let us take a closer look at the accompanying source. We begin with lines 146-160, where we extract the working directory and resource name from the provided URL. Next, we call the function get_gnuplot to acquire the command to execute gnuplot. On lines 173-183, we create a temporary file name that we later use for generating the PNG file.

Next, on lines 184-192, we build our command line and invoke gnuplot using an open pipe, lines 202-211. gnuplot executes the script and renders it to our temporary file as a PNG image. We then take the PNG file produced by gnuplot and, on lines 224-232, stream it back to the requesting client. Once we are done, we delete the temporary file and return success.

Since our bubble will work with any Snorkel based embedded solution, we reuse the server from one of our previous articles to test the bubble.

Running the Example

To run the example:

On Linux

1. Expand the source-binary bundle dashboard.zip
2. Change directories to deployment_directory/bin/Linux
3. Set the library search path variable LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/deployment_directory/lib/Linux

On Windows

1. Expand the source-binary bundle dashboard.zip
2. Open a DOS/command prompt and change directories to deployment_directory/bin/wintel

Start the Server

1. Set the environment variable GNUPLOT to point to the fully qualified path of the gnuplot executable. Note: gnuplot is not included with this article, but you can download its binaries from here (http://sourceforge.net/projects/gnuplot/files)
2. From within the deployment_directory/bin/platform directory, enter the command "fsrv -p 8080" to start the server
3. Launch your favorite browser and enter http://localhost:8080

The index file in deployment_directory/bin/platform references four separate gpng files located in the same directory, displaying them in a two by two table. The files were taken from gnuplot examples and modified for this project.

<html><body> 
<table width="90%" cellpadding="0"> 
<tr><td><img src="finance.gpng" alt=""></td> 
<td><img src="finance2.gpng" alt=""></td> 
</tr> 
<tr><td><img src="iterate.gpng" alt=""></td> 
<td><img src="transparent.gpng" alt=""></td> 
</tr> 
</table></body></html>

You can display each plot individually by renaming the index file (for example, appending a %), or removing it. You will need to stop the server if it is running, prior to performing this action. With the index file hidden or removed and the server started, you can select each file individually from within the browser -- displaying them one at a time.

For example, selecting the file finance2.gpng from within a browser produces the following image:

The gnuscript script that produced the image above is listed below:

set label 1 "Acme Widgets" at graph 0.5, graph 0.9 center front
set label 2 "Courtesy of Bollinger Capital" at graph 0.01, 0.07
set label 3 " www.BollingerBands.com" at graph 0.01, 0.03
set logscale y
set yrange [75:105]
set ytics (105, 100, 95, 90, 85, 80)
set xrange [50:253]
set grid
set lmargin 9
set rmargin 2
set format x ""
set xtics (66, 87, 109, 130, 151, 174, 193, 215, 235)
set multiplot
set title "Change to Bollinger Boxes"
set size 1, 0.7
set origin 0, 0.3
set bmargin 0
set ylabel "price" offset 1
plot 'finance.dat' using 0:3:3:($2>$5?$2:$5):($2>$5?$2:$5) notitle with candlesticks lt 3, \
'finance.dat' using 0:($2<$5?$5:1/0):($2<$5?$5:1/0):
      ($2<$5?$2:1/0):($2<$5?$2:1/0) notitle with candlesticks lt 2, \
'finance.dat' using 0:($2>$5?$2:1/0):($2>$5?$2:1/0):
      ($2>$5?$5:1/0):($2>$5?$5:1/0) notitle with candlesticks lt 1, \
'finance.dat' using 0:($2<$5?$2:$5):($2<$5?$2:$5):4:4 notitle with candlesticks lt 3, \
'finance.dat' using 0:9 notitle with lines lt 3, \
'finance.dat' using 0:10 notitle with lines lt 1, \
'finance.dat' using 0:11 notitle with lines lt 2, \
'finance.dat' using 0:8 axes x1y2 notitle with lines lt 4
unset label 1
unset label 2
unset label 3
unset title
set bmargin
set format x
set size 1.0, 0.3
set origin 0.0, 0.0
set tmargin 0
unset logscale y
set autoscale y
set format y "%1.0f"
set ytics 500
set xtics ("6/03" 66, "7/03" 87, "8/03" 109, "9/03" 130, 
           "10/03" 151, "11/03" 174, "12/03" 193, "1/04" 215, "2/04" 235)
set ylabel "volume (0000)" offset 1
plot 'finance.dat' using 0:($6/10000) notitle with impulses lt 3, \
'finance.dat' using 0:($7/10000) notitle with lines lt 1

It is important to note that right clicking on a gpng file and selecting "Save As" also invokes the gnuplot_gpng function. As a result, instead of downloading the script, the PNG image is downloaded instead.

Conclusion

This project has been on the back burner for quite some time because my original estimates for developing it were large due to a misguided perception of complexity. In actuality, the project turned out to be quite easy.

gnuplot can plot just about anything, and is the ultimate tool for plotting scientific data. I look forward to using it in this form with other natively built solutions.

For those who might be concerned with using Snorkel, it has been in development and testing for the past couple of years, and is a good alternative to other application/embedded server solutions. If you have questions on using the Snorkel SDK, obtaining Snorkel SSL binaries, additional platform support, or enhancement requests, feel free to contact me at wcapers64@gmail.com. Due to unexposed elements within the Snorkel runtime, which are part of a much larger yet to be released project that is currently under development, it is a closed source solution. This is because I have yet to determine whether we will offer the solution provided by the project for free. However, as a CodeProject member, I understand the value of giving back to the community and will continue to offer the basic Snorkel binaries free of charge so that others can benefit from its ease of use and performance. Visit us at http://snorkelembedded.webs.com/ for the latest versions of the Snorkel SDK.