Design and Architecture

axpnumtheory18-Aug-12 8:32

axpnumtheory

18-Aug-12 8:32

You can take a look at the project: http://robotraconteur.net. I have not been publicizing it because the API is still in flux. I don't want to have too many people adopt it and then immediately change the API. Right now the "primary" language is C#. It is the language I am most comfortable with, and have been propagating the changes to other languages. Exceptions are handled by catching any exceptions on the service, saving the exception name and message, and passing the information across the boundary. A container exception is then throw (RobotRaconteurRemoteException) containing the name and message (similar to Java RMI exception handling). A set of built-in exceptions are handled differently that throw a more detailed exception class. You can see the error code in Error.cs. The problem is that the exception names are language-specific, and if someone decides to write a service in a different language the exception names will be different, leading to an annoying design for the clients.

Eddy Vluggen18-Aug-12 10:43

18-Aug-12 10:43

There is no such thing as a common set of errors. The concept does not exist. To make matters worse, people also create their own exceptions.

You can generalize them to a degree, but you can't generalize a solution. That's something that's not only language-specific,.but also platform dependent and even depends on the users' rights. The FileNotFoundException might be uninteresting, or jeopordizing five years worth of data. It might be that I forgot to log on to the remote directory (in which case I'd like to be informed with a message) or I might be a hacker, poking around.

I don't understand what you're trying to achieve, even after reading the introduction.

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

jschell19-Aug-12 6:59

jschell

19-Aug-12 6:59

Thread is getting confused so I am going to rephrase to what I think has been said.

You have a communications library/protocol. You want to allow 'exceptions' to the 'thrown' via that. And you want to support many languages.

That means your library MUST have the following components.
1. A server component.
2. A client component.

In terms of generality 1 and 2 can be different programming languages withing the same enterprise.

So that means that you MUST do the following.
A. Create your OWN protocol method for transmitting an 'exception'. Note that this does NOT mean nor can it mean that you use a language specific exception. Your protocol must NOT use any such exception.
B. Implement your protocol (1) such that is has some common idioms for supporting communication errors. It does that because communication errors are the only common error type that such a library intrinsically knows about.
C. Optionally you provide a way for an implementer to 'extend' your protocol to add their own 'exception' (again this will NOT be a language specific exception.)

AFTER you complete the above then you do the following.
1. Implement your library server code for a specific language so that it catches ALL language specific exceptions. And then it converts them into your protocol as defined above. That and only that is what is sent to the client library.
2. Implement your client library for a specific language so that it receives the protocol 'exception' and translates it into an appropriate language specific exception.

AFTER you have complete all of the above if you have and specific exceptions for specific languages that you want to implement then you use C above to modify the specific library components, perhaps only on the server side but perhaps both sides as well.

If you are clever then you will find that there are ways to define A such that you can use a configuration (rather than code) to add in new language exceptions both in the server and client.

How to introduce Continuous Integration?

Richard MacCutchan19-Aug-12 23:01

Richard MacCutchan

19-Aug-12 23:01

I tried to say this earlier, but your explanation is so much more concise and clear. +5.

One of these days I'm going to think of a really clever signature.

Bernhard Hiller24-Jul-12 22:31

24-Jul-12 22:31

Continuous Integration with its automated builds and test is often seen as an important process/tool in software development, especially in agile enviroments. In our company, we want to introduce it now.
There are some tools available, some of them even freely. And the respective web sites use to claim that they are easy to use. But in a lecture, I heard that setting up CI is not easy at all, a lot of money could be wasted, and it's better to get a consultant for that...
What are your experiences with introducing CI? What was easy, what was complicated? And: what came first - (manual) processes or the tool?

Eddy Vluggen4-Aug-12 22:34

4-Aug-12 22:34

Bernhard Hiller wrote:
What are your experiences with introducing CI?

Never used that term (which is why I initially skipped the question), but have introduced automated builds[^] and testing in three companies. Nothing fancy, motivated to create it by the Spolsky Test[^].

The batch-file gets the latest version of the source-code, builds it, runs the automated tests, runs FxCop, puts the output in a file and mails that every day. Set up once, easy to maintain, no additional costs (except the CPU-time) - hence, very little reason not to do it.

At first, the batchfile would run around midnight, and we'd read the mails in the morning. Currently, it's running at eight O'clock, with the report coming in an hour later. A short list of benefits;

The code is never "broken" for more than 12 hours
Always ready to deliver a "production build"
More confidence in the code-base (none of the external consultants wrote sh*t that breaks my code)
Better maintenance of the tests (since they're run every day)
Statistics on the software; 500+ successful tests every day and the blessing of FxCop are an indication of the quality of your code.

Bernhard Hiller wrote:
a lot of money could be wasted, and it's better to get a consultant for that...

..I'm almost considering on playing the consultant and to sell the batchfile with a random cool name from the BuzzWord-forum Big Grin | :-D

Nah, guess it would be better to have some Rent-A-Coder write a GUI for it first Suspicious | :suss:

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

Bernhard Hiller5-Aug-12 20:54

5-Aug-12 20:54

Thanks for your hints. "Continuous integration" is the common term for that process, see also Wikipedia: http://en.wikipedia.org/wiki/Continuous_integration[^] which lists a number of tools.
The Spolsky test is an interesting idea. Our score is presently about 1. That's the environment where we have to start...

Eddy Vluggen5-Aug-12 23:10

5-Aug-12 23:10

Bernhard Hiller wrote:
which lists a number of tools.

Quite the list; I should be glad I did not waste a Rent-A-Code on that Smile | :)

Bernhard Hiller wrote:
Our score is presently about 1. That's the environment where we have to start...

Is the environment willing to change? If yes, then they might be doubling their productiveness soon.

Good luck Smile | :)

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

pasztorpisti6-Aug-12 4:43

6-Aug-12 4:43

If you want to setup a free CI solution for yourself then you will have to spend some time on reading its documents for the first 1-2 times. I myself played around with buildbot (python) that I hated, and with Hudson CI (java) that is very nice. A bat file is unfortunately hard to monitor (no web interface to look at with a browser), and hard to setup automatic test on submit to your version control system. Currently we use a custom CI server (written in C++), its pure and simple, knows exactly only those few features I listed below in my other post. Over a year we found out exactly what we need and none of the existing solutions provided all of them plus they had a feature bloat we don't need.

Eddy Vluggen6-Aug-12 4:53

6-Aug-12 4:53

pasztorpisti wrote:
A bat file is unfortunately hard to monitor (no web interface to look at with a browser)

BAT

ECHO "Starting unit-test" >/var/www/CIprogress.htm

Custom applications are hard to customize, and a batch-file is easily extended. Anyone who can write a console-application using VB.NET can extend it's functionality. I'll put you a progress-bar in a tray-icon to follow the progress of the batch-file, if required Smile | :)

pasztorpisti wrote:
Currently we use a custom CI server (written in C++), its pure and simple,

Simpeler than what I suggested? Smile | :)

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

pasztorpisti6-Aug-12 5:07

6-Aug-12 5:07

How do you start the script? A CI system can be started from any machine from a browser. You cant start it multiple times by accident. You have a nice page with colors with a list of successful/failed builds. The build starts automatically on submit. You can implement most of these from batch files, but these batch files become uglier then C++ code quickly! Another reason I usually prefer a single binary solution is that script systems (especially heterogeneous ones) are much more error prone.

Eddy Vluggen6-Aug-12 5:57

6-Aug-12 5:57

pasztorpisti wrote:
How do you start the script?

Like any other, although the Windows Task Scheduler might be handy. Or make it a post-build option from the IDE. It's as easy as starting any GUI-less executable.

pasztorpisti wrote:
A CI system can be started from any machine from a browser.

The argument that it does builds for multiple platforms would have been better. It's relative easy to start an executable using ASP.NET.

pasztorpisti wrote:
but these batch files become uglier then C++ code quickly!

..I do not see how anyone could keep a tree with source-code tidy, if they can't even do so with a simple script.

An off-the shelf product does have advantages, you're right. And yes, it does provide some features that would cost some time (and thus money) to implement if you need them. And yes, for larger teams it is recommended to go for an existing product. Especially since no-one will be responsible for maintaining the script, and it'll grow into a number of hacks from various employees. The most decisive limitation here would be the budget.

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

pasztorpisti6-Aug-12 6:09

6-Aug-12 6:09

Choose whatever option you like, and please try them before you make suggestions and tell your opinion. We tried scripts, prepackaged product, and the current option and now we are happier than ever. Choose the option that is suitable for you and minimizes the maintenance work you need to invest! Good luck!

pasztorpisti6-Aug-12 4:34

6-Aug-12 4:34

Our builds are quite complicated, multiplatform, multi-machine, and sometimes even massive data processing is split to separate machines. If your build is a single machine build with a single log file as an output your problem is much easier to solve.
Here is what we find useful in a CI system:
- A web interface through which you can check the result of builds, the log files, and a button which can be used to start or queue up a build.
- scheduled builds
- builds that are triggered automatically when someone uploads new code to your version control system
- sending mails about the results to a few fixed addresses and to those who submitted the latest changes in the version control system since the last successful build (blamelist).

automated tests are not in my list of important things because the config file of our build system is just a set of tasks that are in a dependency graph, each task is a list of commands to execute on one of the machines. automatic testing is just plus one command after running the commands that create the build. Storing a copy of each build to a storage machine is also just an additional command that runs a python script to do the job.

Some companies hire a build engineer for this task, you can also do that if build maintenance is very important. Actually we found CI very useful, especially since our main development platform is Windows and we want automatic tests immediately after submitting C++ code the version control system and it has to compile with visual C++, g++, and llvm compilers as well. Data errors are also often caught by CI.

Bernhard Hiller6-Aug-12 21:19

6-Aug-12 21:19

Thanks for your comments.
Also I find the points you mentioned useful. But I'd include some automated tests (actually lots of regression testing required, but who wants to do them...).

pasztorpisti6-Aug-12 22:12

6-Aug-12 22:12

As I mentioned the build system basically does nothing but runs commands on different machines. An automated test is just a command to run for the build system that can be displayed separately on the web interface. Don't overcomplicate the build system like I did first time! Smile | :)

It just runs your commands, it shouldn't know about your development environment, tests, and so on... If you want something complicated then run a script from your build system (I use python for that). I put a command into the config of our build system if I want to see it in a separate "box"/log file on the web interface, otherwise I just put a set of commands to external script files (if its ok to put their output to the same log file).

BobJanova6-Aug-12 5:19

BobJanova

6-Aug-12 5:19

If you are using a standard language, and you already have automated tests (unit tests, and automatable system tests) which use a standard framework, e.g. JUnit and Ant, then CI is pretty easy to set up. If you're using a less conventional language like Python or Ruby you would probably need to create at least a plugin if not a system yourself.

CI is very valuable in a medium to large project, I strongly recommend you get it set up.

Bernhard Hiller6-Aug-12 21:21

6-Aug-12 21:21

Thanks for your comments.
We are using C#. But automated tests are still o be introduced.

pasztorpisti6-Aug-12 22:17

6-Aug-12 22:17

The build system is independent from the development environment you use, also from your language. I rarely see overly customized build systems (for example I saw one that took screenshots about auto-tests of an opengl software and showed it on the web interface) and it rarely worth the efforts.

BobJanova7-Aug-12 0:24

BobJanova

7-Aug-12 0:24

C# is a standard language, so you should have no problem. We used TeamCity (not free) on a .Net project and everything worked smoothly.

You need to have automated tests for efficient development anyway, so hopefully you can get that done as part of the same push as introducing CI. Does that mean you don't have unit tests? In 2012 that is not really on :P

pasztorpisti6-Aug-12 22:21

6-Aug-12 22:21

CI can be very valuable for small projects too, for example if you start a project that has to be run on many different architectures (PC, mac, xbox, ipad, etc...) then you probably want a dedicated development environment (lets say PC) and you want your crossplatform code to be tested immediately after submitting it to your source control system (or maybe before submitting).

Its very easy to setup CI for a small project if you did setup earlier. For a small project its usually just running a command on a machine or a few machines (one per platform). Plus setting up email etc in the config that is easy.