|
Hello there,
I'm starting up a little personal project that I'm not too sure what the best way to tackle is. I'm hoping someone could give me a few pointers and/or some ideas on questions I should answer before I dig in.
The problem I'm trying to solve to determine file and project dependancies in a big sprawling codebase with tens of thousands of files in it. This codebase has evolved over time and can be quite unwieldy. There are many sub projects inside the build tree and lots of interdependancies between these projects. I want to be able to map these depenencies out.
I found a library that will allow me to moniotr file activity, for example file opens and creates. So my thought is if I run a full clean build and monitor and record all of the file activity of a build I can generate a dependency graph of the entire project. This will also allow me to be able to determine what I should build and in which order when I want to build a tiny piece of the tree. I'm sure there will be other interesting things I can do with this information. I'd also like to try and visualize the entire project and maybe create a file change heat map from it.
My quandry is how best to record the file activity so that I can build a depenecy tree. Ideally I'd like to be able to do this in a multi-threaded way, as our build system can utilize multi-proc machines and have multiple files building at once, and be able to tell which files need to be built before others and which files are grouped in a project and so on.
My current proposed approach is to record all file activity to a logfile and the post process it to generate the depenecy graphs. I'm still a little hazy about all the data that I need to record. I'm currently thinking I'll figure that out as I go when I find I'm missing some important information.
Any pointers or thoughts would be greatly appriciated. 
Thanxx,
Adam
|
|
|
|
|
wouldn't static code analyzation be a better choice by simply parsing all of the includes assuming its a c or c++ program.
or another option to have you project just output the result of the preprocessor in visual studio it will tell you which files it included.
a programmer traped in a thugs body
|
|
|
|
|
Hmm, I'm not sure how hard it would be to create something that would properly process it statically. There is some thinging here that needs to be done around tracking geneology, but once I figure that out I would thing this shouldn't be too hard, but then again I always get blindsided by some of the small details I didn't see when I was thinking about a problem at a higher level. 
Although I don't think all of the files that need to be processed are C/C++ files, there are resource files as other types of files.
And these projects aren't built in VS, they're built in our own build system base off of nmake.
Adam
|
|
|
|
|
Since you are using nmake I'm going to assume you are using gcc while this won't give your resource files it will let you see your source dependencies. How are you using the resource files are they linked into the some type of executable shouldn't you be able to see what you linking in in your make file.
this is from the gcc online manual
-
M
Instead of outputting the result of preprocessing, output a rule suitable for make describing the dependencies of the main source file. The preprocessor outputs one make rule containing the object file name for that source file, a colon, and the names of all the included files, including those coming from -include or -imacros command line options.
Unless specified explicitly (with -MT or -MQ), the object file name consists of the basename of the source file with any suffix replaced with object file suffix. If there are many included files then the rule is split into several lines using \-newline. The rule has no commands.
This option does not suppress the preprocessor's debug output, such as -dM. To avoid mixing such debug output with the dependency rules you should explicitly specify the dependency output file with -MF, or use an environment variable like DEPENDENCIES_OUTPUT (see Environment Variables). Debug output will still be sent to the regular output stream as normal.
Passing -M to the driver implies -E, and suppresses warnings with an implicit -w.
a programmer traped in a thugs body
|
|
|
|
|
Actually we build using the VC compiler, just not inside VS or a VS project. That system doesn't scale well for our needs. While this is an interesting approach I'm still favoring the monitoring of the file activity external to the compiler for serveral reasons:
1. We have C/C++ and C# projects in our tree, and possibly other languages as well that I haven't had to deal with yet.
2. We deal with a number of different building tools that may not have output options like the more mature C/C++ compilers do.
3. Since I have a number of differnt build tools I need to track dealing with each one separately and maintaining the output processing code for handling each tools output sounds like a big task and fragile
Taking a more build tool agnostic approach to gathering this information I feel will make the tool more reliable and take less maintenance work. I won't be trying to track down all of the differnt build tools we use and the necessary command line options to gather the information. And then have to figure out how to process each tools output to gather the information I need.
Adam
|
|
|
|
|
I need some error correction for an integer of 32 bits.
I've found several solutions, like 'reed solomon', 'FEC', 'parity bit'. But all of those are for a lot of bits.
I just need error correction for 32 bits and the error correction should be maximum 24 bits.
Does any of you have any idea of how to clear this problem and which of the error corrections I should use (or even create one myself)?
|
|
|
|
|
Hello,
http://en.wikipedia.org/wiki/Hamming_code - that one is pretty easy, scalable (you decide of length of data chunk, longer chunk - lesser bits will be taken for correction purposes).
Why parity bit is for lot of bits? You can use 1 parity bit for 2 bits of data, for 20 bits, for 200... (of course it will be less effective).
Btw - Hamming codes can correct error bits, not only check if data is not corrupted.
|
|
|
|
|
I see, hamming_code would be great indeed (thank you). But it's only possible for single error correction. In the best case I would like to correct every bit, but that's impossible, but I would like to have the highest efficiency.
I'm now busy with trying parity, horizontal + vertical + diagonal. And then check if the parity's are ok.
But is there another way than the hamming? Because I think this one is not really efficient.
|
|
|
|
|
Hello,
Actually, I think that Hamming is very efficient for what it offers (especially for longer streams, you add 1 bit for every *2 bits of data), if you need more bits corrected - use shorter version etc.
If I have understand you well and by
Deresen wrote: trying parity, horizontal + vertical + diagonal
you mean putting input into matrix and adding parity bits to every row/column, then you won't be able to fix any bits if you will have 2 errors (this is sometimes true, sometimes not, but still, you can't guarantee correcting 2 bits) and the cost of parity bits is much higher then using hamming.
You can also google for Convolutional code, but I don't know much about them, so I can't guarantee it is what you have been lookng for.
|
|
|
|
|
whats wrong with Reed-Solomon?
If I understand your requirements, you need an RS(4,1) code over GF(2^8) or RS(8,2) over GF(2^4), and because the codeword size is small you can use Euclid's algorithm to efficiently compute the key equation rather than Berlekamp-Massey (modified).
check this link out:
http://en.wikipedia.org/wiki/Reed%E2%80%93Solomon_error_correction[^]
|
|
|
|
|
To be honest, I did not really understand the reed solomon error correction.
And I've read this '2 byte errors per 32-byte block', this also means 2 bits per 32 bits. And that's to less for me.
The big problem is that I also have to check the error correction, if that is right. So I have to correct that stream also. Is this a possibility with reed solomon? And could you please give a small example of how the reed solomon works, for instance with a byte?
|
|
|
|
|
Reed-Solomon does provide the ability to detect errors and also correct them, RS is actually quite trivial to understand and implement, RS operates off-of a basis type which is called a symbol. A symbol can be any size from 2-bits and up.
My suggestions were based on two symbol sizes, either an 8-bit or 4-bit symbol size. The 4-bit symbol size would be recommended in your case, as it means 8 symbols per your data block (32-bits). The proposed code of RS(8,2) over GF(2^4) means any two of the 8 symbols can have any number of bit errors (1-4 bits of error/burst error), from this upto both symbols of error can be accurately DETECTED and CORRECTED.
Further more if you happen to know which bits are in error you can increase the correction capabilities 2 fold via erasure correction methods.
If you're familiar with C++ the following library has RS code examples for various bit sizes: http://www.schifra.com/downloads.html[^]
|
|
|
|
|
Thank you very much for your time.
This will give me enough homework for a while.
Let's dive into C++ again.
|
|
|
|
|
What is the expected nature of errors? Is there a significant likelihood of having multiple independent bit errors, or would bit errors more likely be concentrated?
Correcting multiple independent bit errors is very hard; it is clearly impossible to do two-bit correction on 32 bits of data with less than 11 bits of ECC data (using 42 bits to code a 32-bit word, allowing for zero, one, or two errors, there would be 1,765 ways each code word could be represented; a 10-bit ECC would only allow for 1,024).
On the other hand, if all bit errors will be localized in some particular fashion, error detection and correction becomes much simpler. For example, if all errors will be within a single byte, one can store a parity byte along with enough parity information to detect errors in each byte. If a problem is indicated with any byte, the parity byte will allow it to be corrected.
|
|
|
|
|
Hi everyone!
I'm working on a project and i need to do what follows:
i have an array of strings,example:
house3
house2
green23
green.5
H6
H01
G19
G78
..and so on.
I have to extract only the recurrent strings: house,green,H,G.
Any idea about it? (the language is java but doesn't matter, i just need the idea!)
Thanks in advance,
Enrico.
Program your life ^^
|
|
|
|
|
The following should give you some ideas:
http://en.wikipedia.org/wiki/Patricia_tree[^]
...cmk
The idea that I can be presented with a problem, set out to logically solve it with the tools at hand, and wind up with a program that could not be legally used because someone else followed the same logical steps some years ago and filed for a patent on it is horrifying.
- John Carmack
|
|
|
|
|
What are you gonna do exactly ?
Do you want to identify the strings that contain house,green,H,G ?
|
|
|
|
|
not exactly. I have some strings and i have no idea which they are.
I have to identify the recurrent substrings such as house, green...but i don't have them as input..
Program your life ^^
|
|
|
|
|
Some of the problem parameters aren't clear: Do the recurrent strings always start at the beginning of the line? Do you just ignore non-letters (numbers and punctuation)?
If the answer to both of these questions is yes, you can do this:
1. For each line
2. Extract the string of just the letters from the beginning of the line.
3. See if this string is in a hash table. If so, it's a recurrent string.
4. Else insert this string into the hash table.
If the strings don't always start at the beginning of the line, you have to do multiple insertions for each string: the whole string, the substring starting at the 2nd letter, 3rd letter, etc...
|
|
|
|
|
You need to look at using a data structure like a Trie[^], or as was already pointed out Patrcia Trie[^]. The Wikipedia articles should give you enough information to get started on an actual implementation.
Scott Dorman Microsoft® MVP - Visual C# | MCPD
President - Tampa Bay IASA
[ Blog][ Articles][ Forum Guidelines]
Hey, hey, hey. Don't be mean. We don't have to be mean because, remember, no matter where you go, there you are. - Buckaroo Banzai
|
|
|
|
|
When you aren't a mathematician, it's a pain. When you know what its called, there are pages and pages of info. But people rarely put "The DooDingle algorithm calculates the surface area of a dodecahedron cut by a given XYZ plane" on their webpages.
So half the battle is trying to find out what you should be Googling for.
(Incidentally, I'm trying to find a method of locating the bounding triangle in a Delaunay triangulation for any given coord XY, where the vertices of the triangles are a random set of XYZ coords. Then I can work out the Z at any XY, given my grid).
|
|
|
|
|
Try Googling for Delaunay. I have seen it there as well as under "Convex Hull".
Dave.
|
|
|
|
|
Hi! I am working on a graphing application which plots math functions (for example sin(x)). At this stage the program works but in some cases the graph isn't correct. The program calculates the points of the graph and finds the point (x,f(x)). But some calculations return very small numbers an the graph is not correct. In order to fix this I decided also to calculate the limits of the function in the non defined points (for example lim 1/x, x->0). If someone has stumbled upon some articles or knows something about the problem, let me know. (Google only showed me the math definition of the limit, but I couldn't find an algorithm). Thanks!
Still learning...
|
|
|
|
|
It might be a good idea to identify which inputs return wrong answers, then try to find a pattern in the errors. Fixing the underlying bug will produce a more reliable program than adding a patch involving limits.
|
|
|
|
|
Hi......i need help in ASP.NET for vs 2005 to develop an e commerce website for my final year B.Tech. project....plz guide me...........specially how to develop credit cart verification transaction and shopping cart module??????????
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
