It would seem like my thread library is filled with strange, rarely occuring bugs. It's taking me huge amounts of time trying to find and squash them, so I downloaded boost.thread to see how that thread handling library works. I compiled this simple test:
It would seem like I've found the problem with my thread class, it's my condition class that's not working as it is supposed to. I can't see any obvious solution to this problem, but I might find one if I think about it for a while. I've read and re-read the documentation for posix threads without finding out what's wrong, the problem is however best described as a deadlock and could very well be one.
The thread problems might be solved now, so I've encountered a new problem... A huge memory leak, which could be caused by alot of things in my library, so all I can do is start to debug again. It's pretty unlikely that it's the threading part of the library that leaks, which is good since I'm getting tired of debuging those files.
I've narroved the problem down to the select function (my library implementation), it occurs when select spawns more than one thread to select the underlaying sockets. As mentioned before, the memory leak is huge, the size of the process increases with about 15-20 MiB every other second.... It's very confusing.
The select problem is probably solved now, the problem was however not found in the select function at all, but rather in the shared_ptr<> class. All my problems were caused by a small typo in the constructor of the shared_ptr<> class, which meant that it didn't free the memory that it held at destruction.
I'm currently tinkering a bit with the function, adding some features and will later test it some more. When everything seems to be working I'll move on to work with dtpd, testing some locking-patterns and algorithms for thread safety.
select isn't causing me any obvious problems now, I've found and killed som other bugs as well, so I'm currently feeling good about my self. I'm going to look over my dtpd design a couple of times more before actually starting to write the code. On top of that I'm also planing to take one last look at my library, I've realized that the library design will be more or less frozen solid when I start to use it in a bigger project.
I've also realized, while writing the test programs for the network classes, that flexibility and power sometimes should be traded against simplicity. The socket class are not as easy to use as they ought to be, sure they are (almost) as powerfull as real, non-wrapped sockets handled by a C interface, but not nearly as easy to use as their equivalences in other libraries. My ssl socket class has to be rewritten, not much, but it's a bit to hard to use it right now, which is not good.
Ah, progress. I'm finally working on the dtp daemon.
I've started to work on a service class which will make the process that's using it a daemon. Sortof. I'm not sure how it will work in Windows, but it looks promesing for Linux. There's currently a small problem; it crashes at some point by some reason, but I'm debuging it and getting closer to the bug. It would seem like an exception is generated when I try to open a file descriptor to `dev/console'. I've come to suspect that C++ error handling doesn't work well with ::fork().
The policy for service_core is for the moment provide_nothing, which doesn't provide a user defined service as it should (it actually just dead locks). Needless to say, this will be changed in the future.
I've written some code that should make service work in windows too now, I haven't tested it though. I removed the class template for service too, I realized that it was unnecessary to keep it since it did nothing.
int main(int argc, char* argv)
// This is now a deamon/service. Perform whatever work you had planed to do
I've been testing the queue algorithm tonight, it seems to be working fine. I'm moving on to network part now, it's time to test the improved zinara library in action for the first time, and I can't wait to see how it works. I'm hoping that it will be pretty bug-free, I know that it should be, considering that I've tested it extensively.
I'm working with the dtpd object dispatching system right now, I would like the compiler to complain if the I try to place a object in the wrong todo queue, but that would seem impossible with my current dispatching system that relies on virtual functions (runtime) to decide what should be placed where.
There might be a way of doing it with templates, so I'm going to look in to that now.
I haven't really prioritized this blog since I started to keep a changelog file, but here's an update of what has happened since the last post:
I've been working a lot on zinara, the reason is simple; each try I try to use it there's something that isn't working, some huge design flaw which makes the library hard to work with etc. So, these are the changes that I've made to zinara
Bug fixes, get_ip() will now return a correct ip address
Classes completed, ready for testing
Cleaned up, deprecated tests removed.
Comment: Everything seems to be working fine now, the library is installed on my system, seems okay to work with, generates no compile errors and does not seem buggy. I have finally decided how the socket (according to me) is updated, a multiplexer will be used (basically a wrapper around some low-level C function).
Time spent: about 1H.
Directories + source code removed due to license mismatch,
the OpenSSL license is incompatible with GNU GPL.
Added, zinara will use GNUTLS for encryption from now on.
Missing tls implementation right now. (The SSL classes would
have required extensive re-writing anyway due to the major
design changes in the socket part of the library)
Replaced, zinara will use a reactor/multiplexer pattern from now on
Added class templates
Minor bug fixes
Minor design changes
Comment: Unfortunately I've discovered that OpenSSL's License is incompatible with GNU GPL (I do not know why, it's not easy to tell without being a lawyer I guess -> http://www.gnu.org/licenses/license-list.html#OpenSSL ). This means that I will change my secure socket classes so that they use GNUTLS instead, which is released under GNU GPL.
GNUTLS is not yet implemented and I do not think that I'll find the time to write the necessary classes before the first revision has to be turned in either.
Time spent: Including license research about 1 & ½H
Major design changes enforced, changing of template arguments,
re-writing of communication policies, etc. Updated doc tags.
Updated to cope with the new design of the socket template classes.
Added dirty hack to add header guards to the config.hpp file
Comment: The old socket classes worked well in my mind and on GNU/Linux with gcc/g++, but they turned out to be hard to port to other Operating systems and compilers. I have therefore changed the design extensivly which I hope will make it easier to port the library and thereby also dtpd to other systems and compilers.
The new classes are poorly documented and writing new doc tags are not a priority right now, I will probably be able to salvage at least halv of the documentation from the old code when I feel there's time.
Time spent: About 2 & ½H.
Small testprograms for testing socket for both compile-time errors
and run time errors added. (Some older test moved to this directory
Renaming of some files, mostly preparations for the planned design
change of socket classes.
Time spent: About 30-50 min
2007-03-10 -> 2007-03-14
I used about 1H a day to come up with a new socket design and implementing it.
Time spent: Max 6H
Bug-fixes, mainly thread related
Comment: I discovered that my thread safe smart pointers in fact wan't thread safe, there were several lurky race conditions embedded in the code. I hade to re-write all parts of the classes containing critical sections, but the bugs should (hopefully) be fixed now.
Time spent: About 2H.
Formerly known as select_interrupter, minor design changes + renaming
Formerly known as select_timeout, minor design changes + renaming
Comment: Some design changes made, nothing fancy.
Time spent: About 30-40 min.
Added psuedo-support for windows services, needs to be tested and will
probably not compile right away. (Portability issue)
Comment: Added code that will port daemonize(...) to windows, I wrote it using MSDN, to this date it hasn't even been compiled, so it probably contains some minor errors, they shouldn't be hard to fix when I'm forced to port dtpd and zinara to windows though.
Time spent: About 50 min
Classes added to make IO control of the socket as well as setting and
retrieving socket options easier.
Changed IO control and set/get socket option functions, added typedefs.
Comment: Performing I/O control on the sockets and geting and seting socket options was way to hard, I look around a bit and found a brilliant solution in asio. I re-wrote a couple of things and added it to zinara.
Time spent: About 1H.
Bug-fixes: discard algorithm changed, several small typos fixed, segmentation
fault fixed and memory leaks removed.
Comment: The buffer wasn't working at all since I had added an allocator policy, so I had to spend some debuging it and fixing the problems.
Time spent: About 2H. (Segfaults are tricky beasts)
Comment: Just some tinkering in the actual source code, I did however spend some time contemplating on some problems with the dtpd design and managed to solve one consering relationships between objects and workers.
Time spent: About 1H.
And finally, what have I been doing today?
When finally turning back to the dtpd code I've realized that it some major re-writing, the design isn't good enough. So, I've thrown away the latets design and started on a new, some code can be salvaged.
So, I've started to write my report... Just scratched a few things down to get a base to write from. I've been commenting files too, reading about GNUTLS and I've collected some information about mrsw mutex (I have no idea how I am suppossed to turn that into plural, so I'm not even going to try) which are needed in dtpd, from the code I've found this far it should be fairly easy to implement.
I've added windows support for the condition class now, it will probably be error prone and buggy due to the fact that I had to use semaphores. Conditions is a feature that's apperently only available in Windows Vista, I might write support for this some day.
When the condition class is working as it should again, I'm going to use it to write the swmr class (Single writer, multiple readers), it should be fairly simple. The swmr class will then be used to implement the resource lock management system in dtdp. The resource lock management system will be used to make sure that resources are protected with minimal risk of deadlocks in the process.
I'm having some troubles getting started with my report however, it's obvious that the crib sheet handed out by the teachers isn't meant to be used with my kind of project. I should be able to use with some minor modifications though.
I've been writing on my report today, wrote the first page yesterday (but forgot to write about here). I'm almost done with the result part and will finish the discussion part tomorrow. When that's done I'll probably check trough the code and see if there's something that should be fixed before it's turned in. If not, I'll try to make zinara compile with MSVC++, if it doesn't generate to many, all to b****y error messages that is.
I haven't been working on dtpd the last couple of days, but I have been reading the reports assigned to me. They all look good except some minor semantic and syntactic errors. Comming up with 9 good questions/things to improve could become difficult, but probably not impossible.
I've decided that I don't like the autoconf/automake tools. They might be (more or less) *NIX standard, but they're (more or less) horrible too. I'm therefore switching build system for Zinara and DTPD to SCons which seems a lot nicer. It will take some time to do this, but I'm hoping that it will save me (and others) plenty of time in the future.
Still working on the build system change, it's taking lots of time. But I'm hoping it will be worth it. It will make sure that I (probably) don't have to write any special build routines for windows, which would be great.
I could swear that I'd posted another entry here. Anyway, time to sum up what I've done the last couple of days.
The build system switch is done, zinara is now using SCons, it seems to be working nice, which is good. I've also added some documentation, changed the source folder hierarchy, removed a couple of bugs and I'm planning to switch back to work on dtpd again soon.
So, I've been working to get everything working in zinara again. "Adding a simple feature" doesn't exist in my vocabulary any more. Should be done soon though.
I've been working on my report as well, I got some pretty good response on it at last weeks meeting, there are some bugs that has to be fixed. The biggest change will be that I'll add a more detailed description on how dtp works in background, I'll also try to find or make some pictures to go with the parts that might be hard to understand.
Furthermore, I've started to plan my presentation of the project too. It will be nice, and will contain a lot of pictures. Anyway, blogging here is currently wasting time, I need to get back to work.
Time spent: 4H ("Adding a simple feature, shouldn't be a problem", yeah right!)
It feels like I've neglected this blog again (probably because I've been neglecting it). Anyway, Zinara is working again, the new features has been added. I've been working on my presentation of the project the last couple of days though, I've found out that it's harder than I thought it would be to keep everything simple. I think that my current presentation will be easy to understand, but I could very well be wrong. Anyway, I've tried to collect some really nice (some of them even colorfull) pictures to go with the presentation, that should help.
I'm actually a little nervous now, but I'm hoping that everything will work out. I'm trying to keep my presentation as short as possible to, but that's not easy either. Simplicity seems to be adding lenght. Odd. Anyway.... I'll try to remember to write an entry tomorrow about my presentation.
So, I held my presentation today. I have to admit that I have held better speeches, it was pretty good but far from great. I believe that most (hopefully all) people understood what I was talking about though, and that is great. My biggest concern was that my presentation would be to messy and complicated, but it would seem like it was ok.
I've planned to finish my report and make some minor changes to the source code of both zinara and dtpd before handing it in as soon as possible next week.
I've registered a sourceforge.net account today, it took me some time to realize that the ToS and Privacy agreements was okay. I've also finished my report. I'll guess that I'll try to learn enough about sf.net now to be able to create the two project pages that I need.
Still working on the sf.net pages. I handed in the project report with the source of zinara and dtpd today, so this is probably my last entry here. It will be possible to follow the project through sf.net soon, so see you there if you're interested.
Time spent: Can't remember... 2H?
Last Visit: 31-Dec-99 19:00 Last Update: 22-Dec-14 5:55