Add your own alternative version
Stats
683.1K views 4.4K downloads 150 bookmarked
Posted
7 Feb 2005
|
Comments and Discussions
|
|
Could you not use boost::preprocessor to do the same thing?
Make a header file like the one below (mplhelp.h). Each time you include it the value of BOOST_PP_SLOT(1) will be incremented
#include <boost/preprocessor/arithmetic/inc.hpp>
#include <boost/preprocessor/slot/slot.hpp>
#include <boost/preprocessor/cat.hpp>
#ifndef NEW_DISPATCHID_INCLUDED
//
// The first time this file is included assign 0
// to preprocessor variable slot 1
//
#define NEW_DISPATCHID_INCLUDED
#define BOOST_PP_VALUE 0
#include BOOST_PP_ASSIGN_SLOT(1)
#else
//
// On subsequent includes increment the preprocessor variable in slot 1
//
#define BOOST_PP_VALUE BOOST_PP_INC(BOOST_PP_SLOT(1))
#include BOOST_PP_ASSIGN_SLOT(1)
#endif // NEW_DISPATCHID_INCLUDED
///////////////////////////////////////////////
This is OT, but I came across this while trying to use boost::mpl to generate a typelist with a given name but where the types were added to the mpl::list in multiple files - equivalent to the following:
#include <boost/mpl.hpp>
#include <boost/preprocessor/arithmetic/inc.hpp>
#include <boost/preprocessor/slot/slot.hpp>
#include <boost/preprocessor/cat.hpp>
#include <boost/preprocessor/arithmetic/dec.hpp>
#define TYPE(X) BOOST_PP_CAT(X, BOOST_PP_SLOT(1))
#define DECLARE(X, Y) typedef vector<> TYPE(Y)
#define _PUSH(X, Y, Z) typedef push_back<BOOST_PP_CAT(X, BOOST_PP_DEC(Z)), Y>::type BOOST_PP_CAT(X, Z);
#define PUSH(X, Y) _PUSH(X, Y, BOOST_PP_SLOT(1))
using namespace boost;
using namespace mpl;
int main()
{
DECLARE(vector<>, IntList); // equivalent to: typedef vector<> IntList0
#include "mplhelp.h"
PUSH(IntList, int); // results in: typedef vector<int> IntList1;
#include "mplhelp.h"
PUSH(IntList, float); // results in: typedef vector<int, float> IntList2;
//
// Search the typelist for a float
//
typedef find<TYPE(IntList), float>::type IntIter_t;
}
|
|
|
|
|
Hi juggler,
I just tried your your code, and it definitely works. The BOOST_PP macros give me a distinct feeling of black magic, but as long as they compile, right...
There's still a fundamental problem, though; the dispatch ids for RCF need to be generated inside other macros (the RCF_METHOD_xxx macros). You can't embed "#include" statements inside macros, to my knowledge, and I don't see a way around that.
Otherwise this is probably a better method than what I'm using now, by virtue of taking some of the load off the compilers shoulders and putting it on the preprocessor instead.
Jarl.
|
|
|
|
|
I thought the BOOST_PP stuff was evil, too, until I read through the code to see how it worked - and it really isn't too bad.
I know you can't embed #include's inside macros but one technique the PP stuff uses a lot is to #include a macro, e.g. #include MYMACRO. Combining BOOST_PP_SLOT and BOOST_PP_CAT might allow the preprocessor to generate the appropriate filename to include and give you the flexibility you need - or perhaps BOOST_PP_REPEAT would give you what you need.
Nothing against your approach, mind, but I'd be keen to see the library use as much standard (i.e. boost) stuff as possible, especially given that you are the sole developer of the library and that it uses a certain amount of hairy-chested template magic.
The only downside with your approach is that it relies on deeply nested
template parameters, which is often (compile-time) expensive.
I had a link to an article about techniques for avoiding this kind of nesting & recursion but seem to have lost it - it was almost certainly by Dave Abrahams.
Cheers!
|
|
|
|
|
Hi Jarl,
I am very much interested in your framework and was
trying to understand how the whole thing was written.
Do you have any supporting documents that you can
provide or guide me thru this. Because the way it's
been written is drawing a lot of curiousity. So if
you can provide some pointers or documentation that
would be a lot of help.
Thank you,
Nitin
|
|
|
|
|
Hi Nitin,
Sorry there's not very much source level documentation; unfortunately it's likely to stay that way, at least for the current version.
I'm working on a major overhaul of the framework, and when that's ready there will be proper documentation of the user interface and hopefully most of the internals, using something like Doxygen.
That being said, if you have some specific questions you can always ask, and I'll try to explain
Jarl.
|
|
|
|
|
I'm using RCF quite intensively. Using Microsoft VC++ 7.1 I sometimes encounter the error C1204 (compiler limit : internal structure overflow). Clearly RCF is not the only cause, but seems to have some responsabilities (the problem has appeared when I introduced RCF into the project; similar projects without RCF do not raise this error). Sometimes I had to pay more attentions to limit the number of includes (so, let's say the error had a positive side effect), but in other cases the only solution I found was to split the cpp file into simpler ones. One class per file was not enough in some cases.
Did you ever encounter similar problems? Or do you have any compile-time switches (defines) that might help in some way?
Currently, the project that raises the issue uses 8 RCF interfaces with 10 to 20 methods for each.
Anyway, besides this error which is a bit boring, I'm still very happy with the framework.
Paolo
|
|
|
|
|
Yeah I've come across that error, but I don't remember the context. These kind of errors are a real pain to debug, and they seem to arise arbitrarily. One compiler will give an Internal Compiler Error for one reason, another will handle the same thing fine and instead give you an ICE on something else. VC++ 7.1 is usually pretty good; Borland's C++ compiler is a lot worse.
You might try the /Zm switch, although that's meant to alleviate a different error (C1076: "compiler limit: internal heap limit reached"), maybe it might make a difference. Other than that I guess the only thing is to do what you're doing and split up the offending .cpp files.
The metaprogramming that generates dispatch id's for member functions involves a type with template parameters nested some 25 levels deep (25 is the hard coded limit on number of member functions in an interface). If you have a lot of member functions spread across many interfaces, I'm guessing that it might put a relatively high load on the compiler, if you're #include'ing all the interfaces into a single .cpp file.
So RCF-wise, the only thing I can think of is to only #include those interfaces that you need, not automatically #include'ing all of them.
Hope that helps,
Jarl.
|
|
|
|
|
>Performance, as measured in requests/second, is highly dependent on the >serialization protocol, and also on the compiler being used. Before turning to >Boost.Serialization, I used a serialization framework of my own, with which I >could clock around 3000 minimal requests/sec. using Visual C++ 7.1, and 3300 >requests/sec. with Codewarrior 9.0, on a loopback connection on a 1400Mhz, >384Mb PC running Windows XP. GCC 3.2, on the other hand, was far slower. Using >Boost.Serialization, however, I've been nowhere near these numbers; on average, >it's around five times slower.
Have you tried using either www.s11n.net or www.webEbenezer.net
as the serialization framework? I would like to find someone
who isn't working on Boost serialization, s11n.net or
webEbenezer.net that would be interested in comparing the
three.
Brian
|
|
|
|
|
Hi Brian,
I have to admit I haven't used either of the frameworks you're mentioning. After looking over them, I have to say boost serialization looks a lot nicer to use, but I have no idea what a performance comparison would reveal.
There are also different ways of comparing performance. You can construct a single archive, and then serialize/deserialize tons of data to and from it, but in my case, with RCF, thats pretty much irrelevant. Instead, what matters is creating a new archive, serializing/deserializing a small (often < 200 bytes) message, and then deleting the archive, and then doing this repeatedly, on the order of several thousand times per second.
Thats the kind of use in which Boost.Serialization's archives were substantially slower than the ones I had written myself. How s11n or ebenezer would stack up I don't know.
Jarl.
|
|
|
|
|
Hi, Jarl,
Por favor be more specific about why you prefer Boost serialization.
I'm one of the webEbenezer developers so I'm interested in why you say
Boost looks better to use. One thing I wd like to point out is that
we generate a similar function to the one (or two - load/save) Boost
serialization requires you write by hand...
template<typename Archive>
void serialize(Archive &archive, unsigned int version)
{
ar & a & b & c & d & s & m;
}
The similarity is limited, though. The Boost approach requires
n (the number of data members - in this case 6) function calls
to load/save the data. The webEbenezer approach takes 1 call.
Also, the use of one letter member names makes writing the function
easier to write than it wd be in most projects.
Regarding what you were saying about how you use Boost and what
I said about wanting to find someone to compare the different
approaches, I think it wd be important to get input from
those who have developed the frameworks as to whether they had
suggestions on how to improve their horse's results.
Brian
|
|
|
|
|
Hi Brian,
Well, what puts me off immediately about webEbenezer is that it involves a 3rd party code generator and a language of its own (Middle). Its much more complex to tell someone to go to a website and generate some serialization code, than to tell them to write a serialization function a la Boost.Serialization, straight in their source file.
I don't find writing out the member names to be a onus, either, rather the opposite, I want to know exactly what is being serialized.
Anyway, that's just my opinion FWIW, serialization performance has only been noticeable when the server and client are talking to each other on the same machine; network delays will pretty quickly drown out any differences in performance among the serialization frameworks.
Jarl.
|
|
|
|
|
Hey. Thanks for the reply. If I understand correctly, you mean that you aren't
comfortable with the company as a code generation vendor. If that is because
we are small or an unknown quantity, I understand that. But every company is
small and not well known for a while.
I don't think it is more complex to use webEbenezer than to use Boost
serialization. Were you saying that you think the Middle language is complex?
It is your decision if you want to write the serialization function out by hand,
but it seems to me you are getting a raw deal--hand written functions should be
at least as or more efficient than generated functions. I agree with you about
wanting to know exactly what is being serialized, but disagree that Boost has an
edge here. WebEbenezer gives you the source code that does the low level
marshalling. The C++ compilers I'm aware of won't show you the code they
generate.
I agree that network factors will likely dwarf the time it takes to marshall
data, but I'm not comfortable with the implication that the serialization
framework performance is therefore unimportant. (You didn't come right out and
say that, but that is what it sounded like.) There are plenty of applications
(scientific) where performance is crucial and therefore the performance of the
parts matters as well.
Brian
|
|
|
|
|
Hi Brian,
I didn't mean that I was uncomfortable with your company as a code generation vendor, I'm sure you guys do a great job. What I was trying to say is that there's a big ease-of-use gap between an inline C++ solution to serialization, eg as embodied by boost, and an external code-generator solution, with a proprietary description language.
I, and many others, have the the boost distribution on our hard drives, and serialization can then be as simple as adding a few lines to a source file. Later, if the serialized classes are modified, it's completely trivial to modify the serialization functions alongside.
You'll have to correct me if I'm wrong here, but doing the same things with a code generator solution is substantially more cumbersome. Every time I change a serializable class (eg add or remove a serializable member), I'll have to modify some "Middle" code, and then go back to your website and regenerate the serialization code. Then I'll have to merge the result back to my codebase, hoping that I won't lose any of the by-hand modifications that I might have made to the previously generated code.
To me there's a big difference in ease of use, before we even get to the question of performance.
But as you're saying, there are applications where the serialization performance is the bottleneck, and then the ease-of-use factor will take a backseat.
It would be nice to see, black on white, some figures comparing the performance of the various serialization frameworks that are available. I haven't really had time to try anything beyond Boost, and that turned out to be a bit slow for what I'm doing. The serialization framework that I've written myself and included in RCF is fast enough that I don't find improving its performance to be a high priority.
Jarl.
|
|
|
|
|
OK, thanks for clarifying.
- you wouldn't have to modify the Middle code,
- you would have to regenerate the serialization code and
- you wouldn't have to merge anything.
I'll include the start of a thread on c++.moderated that has some
relevant info. Andrea's point about generating the code in the
"final form" is definitely our objective.
1. Hans Guijt Aug 11 2003, 8:49 pm show options
Newsgroups: comp.lang.c++.moderated
From: Hans Guijt <hgu...@xs4all.nl> - Find messages by this author
Date: 11 Aug 2003 20:51:24 -0400
Local: Mon,Aug 11 2003 8:51 pm
Subject: Pointer to member: is this legal C++ code?
Reply to Author | Forward | Print | Individual Message | Show original | Report Abuse
I am attempting to use pointers to members to build a "map" (not in the STL
sense) of a structure. I want to use this map to serialize the structure
automatically.
The tricky bit is that I don't want to repeat the serialisation code on each
struct I want to serialize, since there are quite a few of these (about 480)
and the result is likely to contain errors. Instead I want the compiler to
do the job for me. To accomplish this I've placed the serialisation code and
the array of pointers to members in a base class, and derive the structs I
want to serialize from this base class. In doing this, I end up with
pointers to members that are defined on the base class, but actually
pointing to a member of the derived class.
I have been unable to unearth a reason why this would be illegal, and
"common sense" and experimentation suggests it should work, but common sense
is not always the best guide when it comes to C++. So, is this a legal thing
to do? What theoretical and/or practical problems will I run into? Is there
a better solution?
This is some example code showing the bit I'm concerned about:
class cBase {
public:
int cBase::*MyIntPtr;
};
class cDerived: public cBase {
public:
cDerived ()
{ MyIntPtr = (int cBase::*)(&cDerived::MyInt); };
int MyInt;
};
Thanks for any insight provided,
Hans Guijt
[ See http://www.gotw.ca/resources/clcm.htm for info about ]
[ comp.lang.c++.moderated. First time posters: Do this! ]
2. Andrea Griffini Aug 12 2003, 6:07 am show options
Newsgroups: comp.lang.c++.moderated
From: Andrea Griffini <agr...@tin.it> - Find messages by this author
Date: 12 Aug 2003 06:10:00 -0400
Subject: Re: Pointer to member: is this legal C++ code?
Reply to Author | Forward | Print | Individual Message | Show original | Report Abuse
On 11 Aug 2003 20:51:24 -0400, Hans Guijt <hgu...@xs4all.nl> wrote:
>
The tricky bit is that I don't want to repeat the serialisation code on each
>struct I want to serialize, since there are quite a few of these (about 480)
>and the result is likely to contain errors. Instead I want the compiler to
>do the job for me
.
Allow me a question... did you ever consider to use code
generation ? I do not mean getting crazy in writing magic
template code just to discover that you ended up using a
part on which not all the C++ compilers agree on, but
using an *exernal* code generator, written for example
in PERL or Python. Starting from an input file like:
Employee:
name = string
address = string
level = EmployeeLevel
salary = Money
Department:
code = int
name = string
...
you can generate all the C++ code you need in just a few
lines of a suitable text processing tool.
The only important thing is in this case IMO to be able
to generate the code in its final form (i.e. code that
you do not have to modify manually in the generated source)
because in my experience regenerating will be needed, and
you don't want to lose all the change you made.
Generation can be most probably made part of the make
in your compiling environment (just make the generated
module source code depend on the input file) and so there's
no risk of forgetting to regenerate when you need.
I normally add something like
/**************************************
****************************************
** **
** WARNING **
** Automatically generated code **
** **
** Code generated by xyz.pl: do not **
** edit because any change will be **
** lost at next regeneration. **
** Modify generator instead. **
** **
****************************************
**************************************/
so there should be no risk of coworkers breaking things.
There's nothing that forces you to use ONLY the C++
compiler. IMO there are better tools than the C++
primitive preprocessor and the template machinery for
this kind of use.
You can even generate multiple sources (for example
for a C++ server and a Java client that need to talk).
HTH
Andrea
|
|
|
|
|
Jarl,
I responded hastily last time and want to go back and improve things.
I should have included the Middle code that would be needed...
Message
(MyStruct)
}
The name "Message" is arbitrary. I think that is all the Middle code you
would need. Changes to the definition of MyStruct would get picked up
when regenerating and the Middle code would not need to be changed.
You are right about by-hand modifications being lost, but nobody expects
compilers to be able to merge in modifications that people might make to
object files... I don't think this is different, although it is tempting
because it is source code.
Brian
|
|
|
|
|
Hi Brian,
Thanks for clarifying. I'm concluding that its only necessary to modify the Middle code if I'm adding a new type to be serialized, and that the code generator will automatically serialize all member variables of the type.
The issue of losing by-hand modifications is a pretty serious one, in my opinion, and all the more so because it's an issue that doesn't even arise when using a native C++ serialization scheme. If your code is 100% portable and you know that it never needs to be modified, then OK, but otherwise your clients may well have to (re)modify it every single time they regenerate their serialization code.
IMO, from an ease-of-use perspective, there's not much going for a code generator solution to serialization. But if there are other advantages, such as performance, then it might be worth the extra effort.
Jarl.
|
|
|
|
|
Hey.
>Thanks for clarifying. I'm concluding that its only necessary to modify the Middle code if I'm >adding a new type to be serialized, and that the code generator will automatically serialize all >member variables of the type.
By default all members of a type will be serialized, but there is a way to indicate
you don't want some members to serialized.
>The issue of losing by-hand modifications is a pretty serious one, in my opinion, and all the more >so because it's an issue that doesn't even arise when using a native C++ serialization scheme. If >your code is 100% portable and you know that it never needs to be modified, then OK, but otherwise >your clients may well have to (re)modify it every single time they regenerate their serialization code.
I'm optimistic about the generated code being portable and correct. We haven't needed to
use anything that might be a portability issue. Currently the output is mostly classes,
functions, local variables, for statements, if statements and switch statements. We haven't
had portability problems on the compilers we've tested things on. Compilers have been turning source code into object code that doesn't need to be modified for years and this is not more difficult than that.
One thing we haven't discussed in the ease of use issue is building. Boost
serialization relies heavily on templates. WebEbenezer's approach uses templates
only when users introduce them. I think that building an application with webEbenezer's
stuff would be quicker and simpler than doing the same thing with Boost serialization, but
I don't have numbers to back that up at this time.
Brian
|
|
|
|
|
IMO, letting the serialization framework determine what to serialize is a bad idea. s11n provides a core interface and conventions, and then delegates all actual decisions to functors/proxies/specializations.
----- stephan
|
|
|
|
|
I agree. webEbenezer defaults to marshalling all data
members, but it is possible to exclude members if you
don't want them to be marshalled.
Brian
|
|
|
|
|
Hi, Hi!
It looks like we're all here: i'm the s11n developer. Brian pointed this thread out to me, but i was offline for almost 3 months. Sorry i'm late :/.
You might be interested in reading a fairly objective, fairly detailed comparison i did of Boost and s11n:
http://s11n.net/download/
grab the 1.1.3 manual tarball - it's got .PDF and .HTML formats. i think the section number is 28, but i'm not certain: scan the TOC for Boost.
----- stephan
|
|
|
|
|
> Regarding ... suggestions on how to
> improve their horse's results.
AMEN, Brother!
A tip to software USERS: never underestimate the value of feedback! Psychologists have understood the value of "positive feedback' loops for many years!
----- stephan
|
|
|
|
|
Hello.
It is very interesting library.
I like to extend lib with other data delivery chanels type. Pipes, shared memory, etc. What is the way ? Thank you.
Andrey.
|
|
|
|
|
Hi Andrey,
At the moment there's not really a practical way to extend the delivery mechanism. The library evolved from what was originally a socket-stream type framework, and thats why tcp is the only supported transport
Some time this fall I'll be releasing a new version, and that one will, among other things, make it possible for people to plug in their own transports, like the ones you mentioned.
Jarl.
|
|
|
|
|
Here is something very similar that I wrote a couple of years ago.
My motivations were pretty similar, and I was no fan of SOAP either!
http://www.bugbrowser.com/marshal_getting_started.htm
(web site only works properly in IE - sorry)
There was a C/C++ Users Journal article on it - July 2004.
Take a read - it might give you a few ideas - or maybe not
I provided plugable encoders and decoders.
I provided plugable transports, e.g. raw tcp/ip http pipes etc.
There is a live server that you can reach over the Internet http://www.bugbrowser.com/marshal_live_server.htm
with a demo program to call it.
What you've got here looks great, and it's been tested on more compilers than I tested my marshal:: library (I only used VC7.1 and gcc 3.4).
When C++ 0x supports auto keyword (similar to gcc typeof), we'll probably be able to make the code a lot simpler and make the macros simpler.
|
|
|
|
|
Hi Mark,
Yeah, there are some striking similarities there... Pluggable transports is definitely a good thing, no reason to limit oneself to TCP.
What I would really like to see in C++0x would be macros with variable number of arguments; that would make the interface definition macros look so much nicer.
Thanks for the pointer!
Jarl.
|
|
|
|
|
|
General News Suggestion Question Bug Answer Joke Praise Rant Admin
Use Ctrl+Left/Right to switch messages, Ctrl+Up/Down to switch threads, Ctrl+Shift+Left/Right to switch pages.
|
|