Click here to Skip to main content
13,734,965 members
Click here to Skip to main content
Add your own
alternative version

Stats

69.2K views
2.5K downloads
37 bookmarked
Posted 8 Oct 2015
Licenced CPOL

Debugger for Arduino

, 18 Jan 2018
Rate this:
Please Sign up or sign in to vote.
Source level debugger for Arduino with GDB and Eclipse

Introduction

This article describes source level debugger for Arduino. This debugger can be used to step through your code, place breakpoints, view variables etc. It does not need any modification of the Arduino board or external hardware. It is just a piece of code added to your Arduino program. It works for Arduinos based on ATmega328 microcontroller (tested with Arduino Uno) and also for Arduinos with ATmega2560 or ATmega1280 (Arduino Mega). This debugger has some limitations, as described at the end of the article, but I believe it will be usefull for many people. 

When someone with programming experience from other computer platform starts with Arduino, he or she usually finds it surprising that there is no debugger. After you run your program, there is no way to see what is happening inside other than by printing messages to the serial monitor and/or blinking LEDs. True, it is possible to debug your programs this way, but there are ocasions when stepping through the code or looking at the variables at certain moment can save you a lot of time.
You may think "OK, this is microcontroller; I just have to live with this". But that is not true. Microcontrollers normally have similar debugging capabilities as the "big computers" these days. It is just that the Arduino platform was designed without debugger - there is no debugger interface in the IDE and there is no direct hardware support on the board (in the hardware) either. This is not to say that Arduino is bad. Omitting the debugger is a valid design decision and it is true that for the intended "non-programmer" audience debugger could be just too complicated. But Arduino became so popular that it is now used by almost everyone who needs to do something with microcontrollers, including programmers. And for those people debugger is a useful thing.

There is one misconception about Arduino debugging which seems to be quite common - that if you use a real IDE, for example Eclipse or Atmel Studio, to develop your Arduino programs, you will be also able to debug it as well. It is true that these IDEs contain debugger, but  there is no way for your program to communicate with the debugger . In reality the ways to obtain debugger functionality with Arduino are:

  1. Buy an external debugger device (for example, AVR Dragon) and connect it to your Arduino board. You will need to do small modification of your Arduino board.
  2. Visual Micro debugger plugin for Atmel Studio. I tried it more than a year ago, so please check this for yourself if in doubt, but at that time I did not consider this to be a real debugger. It dit not support stepping through the code and it was based on hidden code inserted into your program before build to communicate with the debugger.
  3. This debugger

What's new?

The January 2018 revision of this article adds description of two new features which greatly improve this debugger:

  • write breakpoints to flash memory
  • load the program via the debugger

With these two features the debugging experience is much like with any standard, hardware-based debugger. With the breakpoints in flash memory the main limitation of the debugger is overcome – the program can now run at full speed with breakpoints inserted. In the older version of the debugger the program ran significantly slower. It is still possible to use the original, so called RAM breakpoints to avoid flash wear.

When using the feature of loading the program via the debugger you can load the program and start debugging with single click. No need to first upload with AVR dude and then start the debug session.

In January 2017 the support for Arduino Mega (ATmega2560 and ATmega1280) has been added. I also found out that it is sometimes possible to use direct serial connection without the TCP-to-Serial converter (proxy). This seems to work with all board on Windows 10, on Window 7 only with Arduino Mega. The tutorial below has been updated to describe the direct connection also.

Background

This section provides some information about how the debugger works. If you just want to start debugging as soon as possible, feel free to skip to the next section.

Modern microcontrollers contain circuitry to support debugging the program inside them. In case of ATmega328 microcontroller used in Arduino Uno, this circuitry is called debugWire. As mentioned above, the problem is that the Arduino platform is not designed to use it and moreover you need an extra piece of hardware to talk with the microcontroller over debugWire, which costs extra money and complicates things.

If we do not want to (or can not) use debugWire, we can use the serial line for debugging. For this to work, the program in the MCU must "talk" with the debugger - our program must contain some code which handles this communication. In principle this is wat you do when you use Serial.print() to output some information to debug your program. But it is not comfortable to stuff your code with the Serial.print() commands. It is better to have a piece of code which talks with the debugger in the background, so you do not need to worry about it. Such a piece of code was invented long time ago, in the times when serial line was the only way to communicate with the MCU.  It is usually called remote stub. This stub has to be able to control the program - to stop it at certain points, read and write memory etc.
What about the other side - the PC, the computer where your IDE is installed? You could probably imagine writing a program for PC which talks to the debugger stub. But there is no need to reinvent the wheel. There is GDB - the GNU debugger, which can handle this and it is supported by many IDEs, including Eclipse.

To summarize, the debugger described in this article is a GDB stub for the ATmega328 microcontroller. It is able to communicate over the virtual serial port provided by Arduino board with GDB debugger on the PC. This way you can debug the program in Arduino using GDB with (or without) some graphical front end (debugger GUI), such us the one included in eclipse IDE.
  

How to use the debugger

Please note that setting up all the things needed for debugging your programs in eclipse may seem quite complicated, especially if you only have experience with the Arduino IDE and start from scratch. If, on the other hand, you are experienced programmer, this should not be too hard.

In this article I will try to give you some idea of how to set up and use the debugger but there is not enough space to cover all the details. There is complete documentation with step by step instructions provided in the download package – see avr_debug.pdf in the doc subfolder.
Here is overview of what needs to be done:

  1. Set up Eclipse IDE to be able to build (and debug) your programs for Arduino. I recommend tutorial in the documentation mentioned above or my earlier article here on codeproject: Creating Arduino programs in Eclipse.
  2. Add debugger library to your Arduino program - this library (driver) is provided with this Article.
  3. Set up the debugger (GDB) in Eclipse.

 

Step 1 - Setting up Eclipse IDE

The goal of this step it to be able to build your Arduino programs in Eclipse IDE. As mentioned above step-by-step instructions are provided in the documentation of this debugger or in my earlier article here.  You can also use other tutorials dealing with building Arduino programs in eclipse.

One important step which is not covered in such tutorials is adding GDB Hardware debugging launch configuration to eclipse. To do this:

In Eclipse go to menu Help > Install New Software…

In the “Work with” box enter the following update site address and press the Enter key:

http://download.eclipse.org/tools/cdt/releases/8.6

After a while the list in the window will display some items.

Expand the CDT Optional Features category and select the C/C++ GDB Hardware Debugging.

Follow the wizard to install this feature and restart Eclipse when prompted.

Note for Arduino Mega

When configuring the AVR eclipse plugin to upload your program (using avrdude), use:

  • For ATmega2560 profile “Wiring” and baud rate 115200.
  • For ATmega1280 profile “Arduino” and baud rate 57600.

Step 2 - Adding debugger support into your program

First, extract the attached zip file into some folder on your computer, for example, c:\avr_debug. Preferably without spaces and/or special characters in the path.

Next, add avr8-stub.c and avr8-stub.h files into your project in Eclipse.
These files are located in the avr8-stub folder in the avr_debug folder.
You can drag the files from your file manager and drop them on your project in the Project Explorer view in Eclipse.
Select Copy files option in the File Operation window which appears after dropping the files.

Open Properties of your project (Alt + Enter) and go to C/C++ Build > Settings.

Select AVR C++ Compiler > Debugging. In the “Debug Info Format” select dwarf-2.

 

Do the same for the AVR C Compiler.

Close the Preferences window with OK.

In your source file include the debugger header avr8-stub.h

#include "avr8-stub.h"

And at the beginning of main() call

debug_init();

TIP: you can also insert call to breakpoint() function into your code and the probram will stop at that line. But you can also insert breakpoints "dynamically", when debugging.

Here is example program to try:

#include "arduino.h"
#include "avr8-stub.h"

void setup(void)
{
    debug_init();
    pinMode(13, OUTPUT);
}

void loop(void)
{
    breakpoint();
    digitalWrite(13, HIGH);
    delay(200);
    digitalWrite(13, LOW);
    delay(500);
}

Try to build the program.

There will be some build errors.  The linker complains about “multiple definition of __vector_1 and vector 18. These are interrupt vectors for the INT0 external interrupt (on pin 2) and interrupt from UART module which signals that a character was received via the serial line. Both these interrupts are needed for the debug driver to work, but are also used by the Arduino software library.

To fix this:

Expand the Arduino folder in your project in Project Explorer in Eclipse and locate the HardwareSerial0.cpp file.

Right-click this file and from the context menu select Resource Configurations > Exclude from Build

In the window which opens select both Debug and Release configurations and click OK.

Now the HardwareSerial0.cpp file will not be built with your program. This will solve the multiple definition for vector 18 (UART), but it also means the Arduino Serial functions will not work. Note that this applies only to this program (project), not to other programs you create either in Eclipse or in the Arduino IDE. You are not modifying anything in your Arduino installation.

Repeat the same procedure for the WInterrupts.c file. That is, exclude this file for built as well.

This solves the multiple definitions for vector 1, but by excluding WInterrupts.c from build, your program cannot use the attachInterrupt Arduino function at all. If you need to use attachInterrupt in your program, you can exclude just the definition of vector 1 in this file. The easiest way is to replace the original file with WInterrupts.c file provided with this article in avr_debug/arduino - see the readme.txt file for details. It will not affect your other Arduino programs. Alternatively, you can modify the file yourself as described in the avr_debug/doc/avr_debug.pdf.

Build the project. There should be no errors now.

Upload the program to your Arduino board.
 

Step 3 - Configuring the debugger in Eclipse

For this section, it is assumed that you already have a program with the debug driver (stub) loaded in your Arduino board. In other words, that you have completed the previous section of this tutorial.

Right-click your project in the Project Explorer in Eclipse.  From the context menu select Debug As > Debug Configurations...

In the Debug Configurations window select GDB Hardware debugging item and click the New launch configuration button in upper left corner of the window. 

This will create new launch configuration under the GDB Hardware Debugging item.
Note: If you do not see GDB Hardware Debugging in the list, you probably haven’t installed this type of configuration. Please see the Setting up Eclipse IDE section above.

Select the new configuration under GDB Hardware debugging to configure its properties. The name of the configuration is based on the name of your project. It is test1 Debug in the picture below.


 

On the righ select Startup tab.

Uncheck (clear) all the boxes (Reset and Delay, Halt, Load image and Load symbols).

 

Switch to the Debugger tab.

In the “GDB command” field enter (or browse to) the path to the GDB executable avr-gdb.exe, followed by the path to your "executable" (.elf) file. 

The path to GDB is [arduino location]\hardware\tools\avr\bin\avr-gdb.exe.

The path to your file can use eclipse variables.
Here is my example for this field:

c:\Programs\arduino-1.6.5-r2\hardware\tools\avr\bin\avr-gdb.exe "${project_loc}/Debug/${project_name}.elf"


Tip: Use the Browse button to select the avr-gdb.exe. Then enter space and paste the following line:
"${project_loc}/Debug/${project_name}.elf".


Check the” Use remote target” box.

Now there are two options for connecting to the debugged program.

  • Direct serial connection
  • Connection via TCP-to-serial port converter (proxy server)

The direct serial connection is easier to use, so I recommend trying this first. I was able to use it on Windows 10 for both Arduino Uno and Mega; on Windows 7 for Mega only. If it does not work, use the connection via TCP-to-Serial proxy.

Note that on Linux you can always use the direct serial connection; just enter the name of the device instead of COMx, e.g. /dev/ttyACM0.

To debug via direct serial connection...

In “JTAG device” select “Generic Serial”.

In the "GDB Connection String" enter the COM port number where your Arduino board is connected, e.g. COM5.

You are now ready to debug. Click the Debug button in the bottom of the Debug configurations window and continue with the Debug session chapter (skip the TCP-to-Serial section below).

To debug via TCP-to-Serial proxy...

Using TCP-to-Serial proxy is less comfortable than direct serial connection, so use this option only if the direct serial connection described above does not work.

In “JTAG device” select “Generic TCP/IP” and enter:
Host name or IP address: localhost
Port number: 11000. Note that the default port number is different, change it to 11000.

Click the Apply button to save the changes, but do not close the Debug configurations window yet.

Now use your file manager (e.g. Windows Explorer) to open the folder where the source code package provided with this article is located. For example, c:\avr_debug.
You should see a start_proxy.bat file in this folder.

Open the start_proxy.bat file in Notepad or other text editor (right click + Edit or Open with...).

Change the number of the COM port in this file. There is this line:

hub4com-2.1.0.0-386\com2tcp --baud 115200 \\.\COM15 11000

Just change the number after COM from 15 to the number of the COM port on your computer where you Arduino board is connected. 

Save and close the start_proxy.bat file.

Run the start_proxy.bat file (double click it).

This will start a convertor between TCP/IP port 11000 used by the GDB debugger (which we configured above) and the serial port to which your Arduino is connected. You should see a console window with some information. This window will be opened all the time during debugging.

You may be prompted to unblock the port by Widnows firewall, allow this.


Now return to Eclipse. We still have the debug configurations window opened.

Click the Debug button at the bottom right of this window.

 

Debug session

When you click the Debug button, Eclipse should ask you if you want to switch to debug view (Perspective). Answer Yes.

You should see the program stopped in debugger, as in the following picture.

You can now step through the code (Step over button in toolbar) to see the LED turn on, etc.

Note that after stepping from the end of the loop, you will find yourself in the Arduino library’s main.cpp file. If you continue stepping, you will get into your loop again. Also, it seems as if the setup function was called again, but this is just discrepancy between the code you see in the C language and the real code generated by the compiler; the setup is not really executed again.

You can use the Resume button to let the program run until it hits the breakpoint we have “hard-coded” at the beginning of loop.

Of course, you can also place breakpoints by right-clicking in the left margin and selecting Toggle Breakpoint from context menu.

If you want to let the program run at full speed, you need to edit the code to remove the call to breakpoint() function. To do so you need to terminate the debug session first. Then rebuild and re-upload the program into the board before connecting with the debugger again. The procedure for changing the program is as follows:

  • Terminate debug session with the red square Terminate button.
  • Close the command prompt window with tcp2com proxy (needed to free the COM port).
  • Change your program, build it and upload to the Arduino board.
  • Start the start_proxy.bat script again.
  • Start debugging in Eclipse (expand the Debug button in toolbar and sclick your debug configuration name).
  • If you receive error when launching, it may be because the project is not selected in Eclipse Project Explorer. Just click on the project in the left window and try again. Or right-click the project, select Debug As > Debug Configurations and start the debug session from there.


If you do not place the call to breakpoint() function into your program, it will run (LED blinking) right after upload. When you connect with the debugger, it will stop at random line; most likely somewhere in the delay() code.  You may see something similar to this:


In the upper window (Debug) there is so called call stack - the “chain” of calls which led the program to its current location. The program is stopped inside the micros() function, which was called from the delay() function, which was itself called from loop() function and so on.

To quickly get into your own code, click loop() in the Debug window (select the loop function). This will display code of the loop function in the lower window. Now you can place a breakpoint, for example, on the digitalWrite(13, HIGH) line, and resume the program. It will stop at the breakpoint.

Flash breakpoints and loading via the debugger

As of January 2018 the debugger has two new features - option to write the breakpoints to flash memory and upload the program into the MCU.

Flash breakpoints

There are two options for implementing breakpoints in this debug driver:

  • Option 1 - store the address at which the program should stop in a variable. Then after executing every instruction of the program compare this variable with the current location of the program (the program counter register, PC). If you find a match, stop there and notify the debugger – let the user know that the program stopped on the breakpoint.
  • Option 2 – replace the instruction at the address where the program should stop with a special instruction which causes “jump” into the debug driver so that it can notify the debugger.

I call the option 1 RAM breakpoints and option 2 flash breakpoints throughout the documentation. In earlier versions of this debugger only the RAM option was available. In current version both options are available. RAM breakpoints are used by default. This can be changed in the avr8-stub.h file,  see AVR8_BREAKPOINT_MODE constant.

The problem with RAM breakpoints is that the program must be stopped after executing every instruction to compare the PC with the addresses of desired breakpoint. This slows down the debugged program considerably. But in fact it is hardly noticeable unless you debug code with longer delays implemented by incrementing / decrementing a counter. 

Flash breakpoints do not slow down the program. The program stops itself on the breakpoint. The drawback is that to use flash breakpoints you need to replace the bootloader in your Arduino, because the debugger needs to communicate with the bootloader to modify the flash memory.  This is described below.

Anyway, I recommend starting with RAM breakpoints first in any case. Only when you are able to debug your program with RAM breakpoints and find out that you can benefit from using flash breakpoints, take the extra step to enable them.

Loading programs via the debugger

The debugger can now receive new version of your program and store it into the memory of the MCU. This feature allows you to start debugging with one click. Normally your program is uploaded to Arduino using separate tool called AVRDude and typical workflow for debugging your program is as follows:

  • Edit the code
  • Build the code
  • Upload the code (via AVRDude)  
  • Click Debug button
  • Debug the program

If you enable the option to upload via debugger, the workflow can be as follows:

  • Edit the code
  • Click Debug button (your code is built automatically before load).

The catch is that you need to replace the bootloader in your Arduino. This is also required for using flash breakpoints so if you do replace the bootloader you get two features for the price of one – you can use flash breakpoints and also load the program via debugger.
To enable loading via debugger, set the constant  AVR8_LOAD_SUPPORT in avr8-stub.h.

Replacing the bootloader

Replacing the bootloader may seem complicated, but in fact it is quite simple and well documented on the internet. But you do need an ICSP programmer or another Arduino board to do this. There are complete instructions in the debugger documentation avr_debug.pdf.

The bootloader needed for flash breakpoints and loading programs is included in the package with this debug driver – see the avr_debug/bootloader/optiboot/debug/optiboot.hex. It is just modified version of the standard Arduino bootloader – optiboot.

Please note that the bootloader is available only for ATmega328 (Arduino Uno and other variants with this MCU). It is not possible to use the flash breakpoints and load via debugger on Arduino Mega and other variants which are not based on ATmega328. I am working on it though…

For advanced users here are the fuse settings needed for the modified bootloader. The size of the bootloader needs to be changed to 1024 words (Set BOOTSZ to 1024 words - bootloader start address 0x3c00). For detailed instructions please see avr_debug.pdf.

Conclusion

I hope this article gave you some idea about the features and use of this debugger for Arduino. It may seem difficult to set up your environment if you are new to eclipse and microcontroller programming in general. You will find detailed step-by-step tutorial in the documentation provided in the download package, see avr_debug.pdf in doc subfolder. 

It is best to move on by small steps, first with simple program written in plain C language and when it works, try a program which contains the Arduino software library. This is how the tutorial in the pdf documentation is organized.

Latest version of the code and documentation is in github repository at: https://github.com/jdolinay/avr_debug.

Limitations of the debugger

I think the debugger in current version can provide features and user experience similar to a hardware debugger without the need to modify your Arduino board and buy the hardware debugger.  However, there are some limitations which you should be aware of.

The Arduino Serial class cannot be used in your program together with the debugger. The debugger uses the serial line to communicate with the eclipse IDE. This may look like a big problem if you write your programs "the Arduino way", that is, print debug messages to serial line. But when you debug with a debugger you usually do not need to print such messages. If you do need it, there is a function debug_message which can be used to send mesages to debug console in eclipse. If you need to send data from your program (for normal operation, not for debugging), then you have to first debug the program without the serial output and then enable the serial output and disable the debugger. Or you can use the SoftwareSerial on Arduino Uno and on Arduino Mega also the other hardware serial interfaces Serial1, 2, etc.

One of the pins with external interrupt function (INT0, INT1,…) must be reserved for the debugger. With Arduino Uno this can be either digital pin 2 or 3 (PD2 or PD3 pin of the MCU). For Arduino Mega there are more options. By default, INT0 pin (Uno pin 2, Mega pin 21) is used, but you can change this by AVR8_SWINT_SOURCE constant in avr8-stub.h file.

As described above, in the default configuration with RAM breakpoints, the program executes at much lower speed when breakpoints are set in the program. This is because the breakpoints are implemented using a little strange feature of the Atmel AVR architecture - there is always one instruction executed after return from interrupt service routine (ISR) before the same or other ISR can be entered again. Thanks to this feature it is possible to single step the program and compare current program counter with desired breakpoint addresses. But having an interrupt triggered after each instruction does slow down the program a lot. This slowness is not a problem in many cases but if you do find it limiting, you can switch to flash breakpoints.

When using flash breakpoints the watchdog cannot be used. Arduino library does not use watchdog so this is usually not a problem. If you need to use watchdog in your application, enable the code which works with the watchdog only after the application is debugged, or use the RAM breakpoints configuration.

Credits

I should mention that it would take much longer to develop this debugger if it were not for some older projects dealing with GDB stub for Atmel AVR. Please see the header of the avr8-stub.h file for more information and links.


The code for this article is also available on github.com at: https://github.com/jdolinay/avr_debug.

History

October 9, 2015 - First version.

January 30, 2017 - Updated version with Arduino Mega support.

January 18, 2018 – Updated version with flash breakpoints and load support in the debugger.

July 12, 2018 - Updated zip archives with bug fixes.

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Share

About the Author

Jan Dolinay
Tomas Bata University in Zlin
Czech Republic Czech Republic
Works at Tomas Bata University in Zlin, Czech Republic. Interested in programming in general and especially programming microcontrollers.

You may also be interested in...

Comments and Discussions

 
QuestionCan this be made to work in VSCode? Pin
Desertgeek14hrs 17mins ago
memberDesertgeek14hrs 17mins ago 
QuestionUnable to see the AVR > AVRDude properties settings (Alt + Enter) Pin
AnyThings7-Aug-18 5:19
memberAnyThings7-Aug-18 5:19 
AnswerRe: Unable to see the AVR > AVRDude properties settings (Alt + Enter) Pin
Jan Dolinay10-Sep-18 2:05
memberJan Dolinay10-Sep-18 2:05 
QuestionGetting stuck at 91% when debugging Pin
11-May-18 19:28
member11-May-18 19:28 
AnswerRe: Getting stuck at 91% when debugging Pin
Jan Dolinay13-Jun-18 22:53
memberJan Dolinay13-Jun-18 22:53 
GeneralMy vote of 1 Pin
saper_219-Jan-18 6:15
membersaper_219-Jan-18 6:15 
GeneralRe: My vote of 1 Pin
Jan Dolinay22-Jan-18 0:39
memberJan Dolinay22-Jan-18 0:39 
QuestionCan we get steps for using your tool in Cmd Line and Pin
21-Sep-17 23:35
member21-Sep-17 23:35 
AnswerRe: Can we get steps for using your tool in Cmd Line and Pin
Jan Dolinay22-Sep-17 2:00
memberJan Dolinay22-Sep-17 2:00 
Questioncmd line and assembly debugging Pin
5-Aug-17 21:51
member5-Aug-17 21:51 
AnswerRe: cmd line and assembly debugging Pin
Jan Dolinay7-Aug-17 0:33
memberJan Dolinay7-Aug-17 0:33 
QuestionCould not determin GDB version Pin
19-Apr-17 8:45
member19-Apr-17 8:45 
AnswerRe: Could not determin GDB version Pin
Jan Dolinay20-Apr-17 23:55
memberJan Dolinay20-Apr-17 23:55 
GeneralRe: Could not determin GDB version Pin
14-May-17 6:35
member14-May-17 6:35 
Questionparameters values are wrong Pin
impeham14-Feb-17 7:39
memberimpeham14-Feb-17 7:39 
AnswerRe: parameters values are wrong Pin
Jan Dolinay15-Feb-17 2:41
memberJan Dolinay15-Feb-17 2:41 
GeneralRe: parameters values are wrong Pin
impeham15-Feb-17 3:21
memberimpeham15-Feb-17 3:21 
GeneralRe: parameters values are wrong Pin
Jan Dolinay16-Feb-17 0:06
memberJan Dolinay16-Feb-17 0:06 
GeneralRe: parameters values are wrong Pin
impeham16-Feb-17 13:44
memberimpeham16-Feb-17 13:44 
Questionstart debug gives an error Pin
impeham10-Feb-17 23:01
memberimpeham10-Feb-17 23:01 
AnswerRe: start debug gives an error Pin
Jan Dolinay12-Feb-17 22:25
memberJan Dolinay12-Feb-17 22:25 
GeneralRe: start debug gives an error Pin
impeham13-Feb-17 23:05
memberimpeham13-Feb-17 23:05 
GeneralRe: start debug gives an error Pin
Jan Dolinay14-Feb-17 2:18
memberJan Dolinay14-Feb-17 2:18 
GeneralRe: start debug gives an error Pin
impeham14-Feb-17 3:09
memberimpeham14-Feb-17 3:09 
GeneralRe: start debug gives an error Pin
Jan Dolinay14-Feb-17 22:12
memberJan Dolinay14-Feb-17 22:12 

General General    News News    Suggestion Suggestion    Question Question    Bug Bug    Answer Answer    Joke Joke    Praise Praise    Rant Rant    Admin Admin   

Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.

Permalink | Advertise | Privacy | Cookies | Terms of Use | Mobile
Web04-2016 | 2.8.180920.1 | Last Updated 18 Jan 2018
Article Copyright 2015 by Jan Dolinay
Everything else Copyright © CodeProject, 1999-2018
Layout: fixed | fluid