High performance file format design

Question

3.00/5 (1 vote)

See more:

I design a file format that contains four blocks:Header Block,Data Block,Event Block,Remark Bolck recently.This file format is used in data collection situation.
The Header Block is fixed length.The Data Block is increase in data collection progress.The Event Block,Remark Bolck as also.
|Heder |
|Collect Data Row 1 |
|Collect Data Row 2 |
|Collect Data Row 3 |
|...... |
|Collect Data Row n |
|Event Data Row 1 |
|Event Data Row 2 |
|Event Data Row 3 |
|...... |
|Event Data Row n |
|Remark Data Row 1 |
|Remark Data Row 2 |
|Remark Data Row 3 |
|...... |
|Remakr Data Row n |
Because long process of data collection,the collection file is so big that must use mapview to acess or read.
Assume that the Data Block allocation 64K size space, and data collection has filled the space.If you continue to write the data will cover the Event Block.How to design a file or map in order to ensure they are stored contiguously in a file?
My english is poor, please forgive me.

Posted 13-May-12 17:24pm

johnye1982

Add a Solution

2 solutions

Add a Solution

Add your solution here

Treat my content as plain text, not as HTML

Preview 0

…

Existing Members

Sign in to your account

...or Join us

Download, Vote, Comment, Publish.

Your Email
Password
Forgot your password?

Your Email
This email is in use. Do you need your password?
Optional Password

I have read and agree to the Terms of Service and Privacy Policy
Please subscribe me to the CodeProject newsletters

When answering a question please:

Read the question carefully.
Understand that English isn't everyone's first language so be lenient of bad spelling and grammar.
If a question is poorly phrased then either ask for clarification, ignore it, or edit the question and fix the problem. Insults are not welcome.
Don't tell someone to read the manual. Chances are they have and don't get it. Provide an answer or move on to the next question.

Let's work to help developers, not make them feel stupid.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

Peter_in_2780 · Answer 1 · 2012-05-13T17:37:00

Solution 1

Why not use separate files? You can keep related information together either by a file name convention (Head0001, Data0001, Evnt0001, Rema0001, etc) or grouping them by directory structure.

Peter

Posted 13-May-12 17:37pm

Peter_in_2780

Comments

Sergey Alexandrovich Kryukov 13-May-12 23:56pm

[OP commented:]

If the file is stored separately, you need to maintain the connection relationship of the document.File separate storage does not meet the security requirements.

Sergey Alexandrovich Kryukov 13-May-12 23:57pm

Aha, and a single file meets security requirements... who told you so?
--SA

Sergey Alexandrovich Kryukov 13-May-12 23:59pm

Not a bad idea, my 5, but serious analysis of the design needs more detailed information, which OP did not share with us.
--SA

johnye1982 · Answer 2 · 2012-05-13T18:05:00

Solution 3

For a single file can be digitally signed. However, multiple files how to ensure that the connection relationship of the file has not changed?

Posted 13-May-12 18:05pm

johnye1982

Comments

Peter_in_2780 14-May-12 0:15am

You can include some unique identifier (like a GUID) in each file of a related set, and use that as part of the signed data. That way, your digital signature mechanism can protect not only each file but the relationship between them.