C / C++ / MFC

11-Dec-08 2:33

When reading a binary file, and you have no clue what the delimeter may be, since you only care about the raw number of bytes you read, and yes there are thousands of \n's in the file for some reason.

Is there a single function that can do that without specifying what the delimeter is or do I have to read this character by character?

Don't forget to vote if the response was helpful

Sig history

"dad" Ishmail-Samuel Mustafa

Unix is a Four Letter Word, and Vi is a Two Letter Abbreviation

"There is no wealth like knowledge, no poverty like ignorance" Ali Ibn Abi Talib

Luc Pattyn11-Dec-08 2:36

Luc Pattyn

11-Dec-08 2:36

there are functions that ignore the data completely. What is it you are using right now?
Maybe you could show some code...

Smile | :)

Luc Pattyn [Forum Guidelines] [My Articles]

Fixturized forever. Confused | :confused:

Mustafa Ismail Mustafa11-Dec-08 2:46

11-Dec-08 2:46

<br />
char* buffer = new char[settings.getAccessesToSimulate()];<br />
char* fname = settings.getFileName();<br />
<br />
ifstream binFile(fname, ios::in|ios::binary);<br />
<br />
if(!binFile)<br />
{<br />
   //handle error code<br />
}<br />
<br />
binFile.read(buffer,settings.getAccessesToSimulate());<br />
<br />
binFile.close();<br />
<br />

I also tried, get and getline after that with the same results. After this little beauty runs, I get buffer = "gsA" which is incorrect since I'm giving it 1000 characters to read. Opening the file manually (in windows, not linux where its supposed to be) shows me that there's an NUL right after those three characters (they're actually hex).

Don't forget to vote if the response was helpful

Sig history

"dad" Ishmail-Samuel Mustafa

Unix is a Four Letter Word, and Vi is a Two Letter Abbreviation

"There is no wealth like knowledge, no poverty like ignorance" Ali Ibn Abi Talib

Stuart Dootson11-Dec-08 2:58

11-Dec-08 2:58

Mustafa Ismail Mustafa wrote:
After this little beauty runs, I get buffer = "gsA" which is incorrect since I'm giving it 1000 characters to read. Opening the file manually (in windows, not linux where its supposed to be) shows me that there's an NUL right after those three characters (they're actually hex).

But what is there in buffer **after** the NULL? You're displaying buffer as a C string, so printf (or whatever) sees the NULL and presumes that's the end of the data in buffer. It probably *isn't*. The trouble is that the C string semantics get assumed for any char*, but those aren't the semantics you want.

You probably ought to iterate through buffer with a for-loop or something to see the data after the NULL.

Mustafa Ismail Mustafa11-Dec-08 3:04

11-Dec-08 3:04

That's what I did. Buffer only contains those 3 characters. I know the file has 4,000,000 bytes of data that I need extracted.

I'll be more than happy to change the implementation to something else. I'll just need to be pointed in the right direction Smile | :)

Don't forget to vote if the response was helpful

Sig history

"dad" Ishmail-Samuel Mustafa

Unix is a Four Letter Word, and Vi is a Two Letter Abbreviation

"There is no wealth like knowledge, no poverty like ignorance" Ali Ibn Abi Talib

Stuart Dootson11-Dec-08 4:32

11-Dec-08 4:32

I've just built your code into a small executable and read a file containing the following text:

48 65 6C 6C 6F 00 48 65 6C 6C 6F

That's two instances of the word 'HELLO' separated by a NULL. I looked at the contents of buffer using the 'Examine Memory' feature of WinDbg and it contained the 11 bytes shown above. That's what you wanted to achieve, no?

Mustafa Ismail Mustafa11-Dec-08 4:41

11-Dec-08 4:41

Precisely. So why is my buffer reading 4 bytes (the 3 characters and the terminator) before exiting? Any ideas?

Don't forget to vote if the response was helpful

Sig history

"dad" Ishmail-Samuel Mustafa

Unix is a Four Letter Word, and Vi is a Two Letter Abbreviation

"There is no wealth like knowledge, no poverty like ignorance" Ali Ibn Abi Talib

Stuart Dootson11-Dec-08 6:50

11-Dec-08 6:50

One other thing I can think of - what's the value of settings.getAccessesToSimulate()? We've shown that your file reading code is correct - but are you reading the right number of bytes?

Mustafa Ismail Mustafa11-Dec-08 10:34

Re: Ignoring \n in a binary file during reading [modified]

11-Dec-08 10:34

Sorry for the late reply.

That was the first thing I checked and after I did get it fixed ( I wasn't making a constant reference of all things!) the number currently I'm using is 1000 though it could range from 0 to 1,000,000.

This is on Fedora if it makes any difference though it should not.

Don't forget to vote if the response was helpful

Sig history

"dad" Ishmail-Samuel Mustafa

Unix is a Four Letter Word, and Vi is a Two Letter Abbreviation

"There is no wealth like knowledge, no poverty like ignorance" Ali Ibn Abi Talib

Stuart Dootson11-Dec-08 12:53

11-Dec-08 12:53

I've just tried your code on OS X - worked fine there as well.

One last suggestion - you could try using the C file handling functions (open, read, close), like so:

int fd = open(fname, O_RDONLY);
const int nBytesRead = read(fd, buffer, 1000);
close(fd);

That at least tells you how many bytes it's read?

[edit]In Visual C++, this probably ought to be:

int fd = _open(fname, _O_RDONLY|_O_BINARY); const int nBytesRead = _read(fd, buffer, 1000); _close(fd);

VC++ uses the "ISO C++ conformant" names rather than the POSIX API names...ho-hum.

[/edit]

On the same note - std::ifstream has a method called gcount that tells you how much the last read actually read:

binFile.read(buffer, 1000);
std::cout << binFile.gcount() << std::endl;

Just tried both of those on OS X - both give the expected results with files that contain embedded NULLs.

modified on Thursday, December 11, 2008 7:00 PM

Mustafa Ismail Mustafa11-Dec-08 21:03

11-Dec-08 21:03

This is insane.

After restarting the computer, things worked perfectly, the way they should have in the first place. Why would something like this happen?

Thanks much Stuart, you and rest helped, and I got to (re)learn a few more things.

Cheers

Don't forget to vote if the response was helpful

Sig history

"dad" Ishmail-Samuel Mustafa

Unix is a Four Letter Word, and Vi is a Two Letter Abbreviation

"There is no wealth like knowledge, no poverty like ignorance" Ali Ibn Abi Talib

Stuart Dootson11-Dec-08 21:11

11-Dec-08 21:11

Mustafa Ismail Mustafa wrote:
After restarting the computer, things worked perfectly

Don't you hate computers sometimes - as recalcitrant as a small child Big Grin | :-D

Mustafa Ismail Mustafa11-Dec-08 21:44

11-Dec-08 21:44

Stuart Dootson wrote:
Don't you hate computers sometimes

With extreme prejudice. I have a computer that I'd like to introduce to a baseball bat at 90mph. One day I will, instead of relegating it to our family "museum".

I have a slightly different thing now. Bear with me, its been a few years since I've worked this "heavily" with C++.

A different binary file, is being read and this time, I need to clump every 4 bytes into one structure and this structure would be a member in an array of its type. Suggestions?

Don't forget to vote if the response was helpful

Sig history

"dad" Ishmail-Samuel Mustafa

Unix is a Four Letter Word, and Vi is a Two Letter Abbreviation

"There is no wealth like knowledge, no poverty like ignorance" Ali Ibn Abi Talib

Stuart Dootson11-Dec-08 23:11

11-Dec-08 23:11

Mustafa Ismail Mustafa wrote:
A different binary file, is being read and this time, I need to clump every 4 bytes into one structure and this structure would be a member in an array of its type. Suggestions?

Pffff - kind of depends on a lot of things Smile | :)

, like:

Does the file have the same endian-ness as the computer you're working on?
Do you know how many elements the array has?
What format is the 'structure' you talk about?

Possible solutions include:

Interpreting the file byte-by-byte, to read into the structure
Read the file a structure (four bytes at a time), assuming that the structure can be represented as four contiguous bytes in memory. I'd use #pragma pack to do that, with a static assert[^] to ensure that the structure really takes up four bytes in memory. If endian-ness is an issue, then you can reverse bytes at this point.
Read the whole array in, if you know how many members it has. To ensure that padding doesn't get in the way, you could read in the correct number of bytes (sizeof structure * element_count), then use a byte pointer to step through the bytes, casting the pointer to a structure pointer when accessing the bytes

Yes, I've done this stuff a fair bit. From reading and interpreting executables (not just PC ones - that's where the different endian-ness has come in) through decoding data received over a serial link to reading and interpreting gigabyte-sized data files logged when testing embedded systems.

I generally use memory-mapping so I don't need to bother with reading the files myself. In the case of gigabyte size files, this means mapping different bits of the file at different times, 'cause there's not enough RAM/address space to map the whole thing at once.

I find that using custom iterators (with the help of the Boost Iterator library[^]) helps, as you can abstract the conversions/file access/pointer incrementing into the iterator, allowing you to concentrate on the structures you're actually reading from the file. This especially helped with the big memory-mapped file thing, as I could hide the mapping/unmapping of the file, and (in effect) get a random access iterator over the file that meant I never had to deal with the file *as a file*. That was sweet - maybe I ought to write an article about it.

In general, I'd advise using layering in your application's architecture, so you deal with the details of the binary file *in a single, self-contained place*, allowing you to deal with higher-level concepts withouyt worrying about the lower-level ones.

Mustafa Ismail Mustafa12-Dec-08 1:00