The whole problem is so awkward and not easy to implement, so it's hard to give a comprehensive advice. However, the idea of implementation of the database storage might be a useful exercises. (Perhaps one goal would be to develop the appreciation of existing database systems. :-))
The problem is: it looks like you try to use a plain text file where a line represents some record. Generally, it's a very bad media for the database storage, because the lines can be of different lengths. Imagine you need to modify one record close to the beginning of the file. If the modified record makes different length of the line, you have to rewrite the rest of the file, because each and every character will be shifted. There is nothing you can do about it: in general case, you have to
read the whole file, and
write it all back, or a big part of it, 1/2 of the file size on average. You cannot even skip any part of the file on reading, because you never know the length of each line.
One solution is to create another file. You can call it "index file". In its simplest form, it can be used to store the position of each line in the main file, as well as the length of each record. Moreover, you may need to have several indices. For example, one index would be sorted by one field, another by another field. Then the main reading will be done via peeking the line at given position. However, the problem with modification will remain the same: you will need to rewrite the rest of the file; also, you should keep integrity between index files and main file.
Note that if this is a kind of relational database, we are only discussing a single table represented by the file. In relational model, you would need to have multiple table, keys, foreign keys, all that stuff.
Anyway, another approach for a single table would be fixed-size records. Let's say, in a table metadata you setup fixed sizes of all data types. It is especially painful for strings: for each attribute, you will need to have length limit. With fixed-size lines, you don't even need the
end-of-line characters. You just peek a record by multiplying the record number by record size, go to this position in file and read/write it, without rewriting the rest of the file. But the limitation of string lengths is draconian…
Well, just some food for thought…
—SA
Submit an article or tip