Palm databases - how to persist data on a Palm hand held

• Download source files - 14.2 Kb
• Download demo project - 3.39 Kb

Introduction

Welcomed this, my third article on palm development. Now that we have covered the basics and can create an application with some GUI components, the next important thing is to be able to persist data. The palm application philosophy is simple. When an application is started, it should be in the State that it was last left, giving the illusion of a multithreaded system, were in fact only one thread can run at a time. In order for this to be possible, as well as in order for just about any useful application to be written, we need to store data. Because the palm has no external storage mediums, data is stored in memory, in area is known as databases. I am sorry to report that this is somewhat of a misnomer, and does not in any way imply facilities such as SQL or relational tables. Instead what you get is a block of memory which is associated with your creator ID and the name, from which you can select data by either its array style index, or an ID which is guaranteed to be unique in the context of that device, and which would have no relation to the ID returned if the same application were run on a different device.

Creating a database

The first step of course is to create a database. The variable which equates to a handle to a database is a DmOpenRef. You should declare one of these as a global in your main header file for each database that you need to keep open. The call to create a database looks like this.

Err err = 0;
err = DmCreateDatabase(0, "BugSquBugsDYTC", 'DYTC', 'DATA', false);

DYTC is my unique creator ID, it stands for Dytech, which is the name of my employer. You should have previously registered your own ID on the www.PalmOS.com web site, and you should substitute that as the third parameter. The first parameter represents the card number on which to create a database, and it is almost universally 0. The second parameter is a name for a database, you will note that I have used my creator ID within the name, which palm recommend doing in order to ensure that your database name is unique. The fourth parameter is the type of database, and can be any for character constant. You may not have noticed this, but in fact that creator ID AND the type are actually 32-bit numbers, which are generated from the four 8-bit char values which we provide. In theory one could create a number of databases for an application which have the same name, and differ by type. However, in my experience, I have found that unless the type is DATA, your data base will appear within the palm as a file separate from your application, which I regard as undesirable. The final parameter is a ball in value, which specifies if this is a resource database. For our purposes, we will only be creating record databases, therefore this parameter will always be false.

Opening a database

At this point, we still do not have an open database and we have not used our DmOpenRef. We're now going to correct both of these anomalies. There are two ways to open a database. The first is as follows:

DmOpenRef myRef;
DmOpenDatabaseByTypeCreator('DATA' , 'DYTC', dmModeReadWrite);

The parameters are the database type, the creator ID, and the mode in which to open the database. This seems to be the wide almost universally preferred by both the official palm documentation, and the third party books that I read. Why this is so is a mystery to me, because it does not use the name which we have so carefully crafted to be unique for this particular device, and instead requires that we either register a creator ID for every database we want to create, or that we create databases with arbitrary types of our own invention, and as a result (at least in my experience) have databases which our users can delete apart from application. The method which I prefer is a little more long-winded, but I believe it is superior nonetheless. It looks something like this:

Err err = 0;
LocalID dbID = DmFindDatabase(0,"BugSquBugsDYTC");

gDB = NULL;

if (dbID > 0) gDB = DmOpenDatabase(0, dbID, dmModeReadWrite);
if (!gDB) 
{
	err = DmCreateDatabase(0, "BugSquBugsDYTC", 'DYTC', 'DATA', false);

  	if (err == 0)
	{
		gDB = DmOpenDatabase(0, DmFindDatabase(0,"BugSquBugsDYTC"), dmModeReadWrite);

The first API call we make it is DmFindDatabase, which takes the card number and the database name and returns a LocalID, which I presume to be a 32-bit number, on the basis that when I first tried it by used a UInt16 and it did not work. This ID when it is returned can be passed to DmOpenDatabase, which takes the card number again,our LocalID, and the open mode required. If it fails, the return value is null and so we continue to create our database both in the distance where it cannot be found, and where it cannot be opened. Assuming the database is successfully created, we then open it. Note that we need to call find database again, as the value returned the first time was invalid, all we would have not reached this point.

Creating records

So at this point we have a database on a palm, and a handle to it. Quite obviously the next step is to create some data records with the database, in particular it is likely that our newly created database should have some default values within it. The sample application is called Bug Squasher. It was written as a learning exercise, and in theory is designed to be part of a bug tracking system, where the desktop system updates a list of applications which can have bug reports, and the Palm creates bug reports which can then be uploaded to the main system. The simpler of the two databases simply stores a list of names, which equate to projects for which we can report bugs. The two default values are 'misc' ( for bugs on systems not in the Palm ) and 'BugSqu' ( so we can report bugs on the bug tracking system itself ). The functions we will need to create a record and fill it with data are DmNewRecord, which takes three parameters ( the database to operate on, the index of the record to create, and a length for the record ), and DmStrCopy, which takes a Char * ( derived from our record handle, as we will see ), a start position, and another Char *, which we will copy. Note that I capitalise the word Char, this is a common way for PalmOS to create it's own types, a Char is the type used by Palm, as opposed to char. That they are the same thing is immaterial, Palm create their own types to ensure all types are the same size across platforms. The other functions we will use are MemHandleLock, which returns a void *, and which we call on our new database record, casting the result to a Char *. Finally, MemPtrNew is the Palm equivelant of malloc, and is used to create our string, StrCopy is the equivelant of strcpy and is used to place a value in our string, and MemPtrFree is used to free the string we created. Here is the code:

UInt16 index = dmMaxRecordIndex;

MemHandle h = DmNewRecord(gDBProj, &index, 5);

if (!h)
    err = DmGetLastErr();
else 
{
    Char * s = (Char *) MemHandleLock(h);
    
    Char * pChar = (Char*)MemPtrNew(5 * sizeof(Char*));
    StrCopy(pChar, "Misc\0");
    DmStrCopy(s, 0, pChar);
MemPtrFree( pChar);
    MemHandleUnlock(h);				
}

index = dmMaxRecordIndex;

h = DmNewRecord(gDBProj, &index, 6);

if (!h)
    err = DmGetLastErr();
else 
{
	Char * s = (Char *) MemHandleLock(h);
    
    Char * pChar = (Char*)MemPtrNew(6 * sizeof(Char*));
    StrCopy(pChar, "BugSq\0");
    DmStrCopy(s, 0, pChar);
MemPtrFree( pChar);
    MemHandleUnlock(h);				
}

dmMaxRecordIndex is a Palm defined constant which causes our record to be created at the end of the database. Note we need to set it twice, as when we pass it into DmNewRecord, it is set to be the actual index of the created record. It's not shown here, but DmReleaseRecord should be called to release the record as soon as it is no longer needed.

Well, that was easy, wasn't it ? But because the databases are just an area of memory and offer no real facilities beyond locking a record in terms of accessing items within a record, what do we do when we want to store more complex data ? The answer is to create a struct that defines our datatype. For example, if we had a strict that looks like this:

typedef struct {
  DateType DateEntered;
  UInt8 nBugSeverity;
  UInt8 nBugType; 
} BugRecord;

then we could simply use sizeof to get the size of our struct, and make our new records that size, and then write them using the DmWrite API, which has this prototype - Err DmWrite (void *recordP, UInt32 offset, const void *srcP, UInt32 bytes), where the void * is a pointer to our record, the offset indicates how far into the record to start writing, srcP points to the data, and the UInt32 indicates the number of bytes to write. An example would be as follows:

Char *s;
UInt16 offset = 0;
s = (Char *) MemHandleLock(DBEntry);
DmWrite(s, 0, br, sizeof(BugRecord));
MemHandleUnlock(DBEntry);

I'm sure that many of you spotted a problem. This isn't the full bug record, it does not contain a string. If you have to store data which does not involve strings, or the strings are of a fixed size, then life is easy. However, as I often say, space on the Palm is at a premium, and so if we have strings in our data, we must take steps to ensure that each record takes up the smallest amount of space possible. In order to do this, we need to undergo a process commonly referred to as 'packing' our struct into a record. In order to do this we create two struts, one which contains Char *'s, and the other which simply contains Char arrays with a length of 1. Here are the real structs from the Bug Squasher demo application.

typedef struct {
  DateType DateEntered;
  UInt8 nBugSeverity;
  UInt8 nBugType;   
  char * szBugDesc;
} BugRecord;

typedef struct
{
  DateType DateEntered;
  UInt8 nBugSeverity;
  UInt8 nBugType;   
  char szBugDesc[1];
} PackedBugRecord;

In this case we only have one string, but the example holds for as many strings as you would like. Because the only way to find the value of the nth string is to unpack all n strings, you should always place the most important strings at the top of the string list, and all non string information before that. Storing variable length records causes us to need another function, which we will use within a helpful function we write, called PackBug. The API function is called MemHandleResize, and it takes a record and a new length and like most Palm functions returns an Err which is 0 if there were no problems. Here is our PackBug function.

void PackBug(BugRecord *br, MemHandle DBEntry)
{
	UInt16 length = 0;
	Char *s;
	UInt16 offset = 0;
	length = (UInt16) (sizeof(BugRecord) + StrLen(br->szBugDesc) + 1);

	if (MemHandleResize(DBEntry, length) == errNone)
	{
		s = (Char *) MemHandleLock(DBEntry);
		DmWrite(s, 0, br, sizeof(*br));
		DmStrCopy(s, OffsetOf(PackedBugRecord, szBugDesc), br->szBugDesc);
		MemHandleUnlock(DBEntry);
	}
}

As you can see, we calculate the length of the record by the size of the struct and the length of the string, plus one for the null terminator. Obviously if you are storing more than one string then will need to store enough space to null terminate them all. Note also that the Palm API provides a function called OffsetOf, which allows us to find the offset within a struct of the string in question. The C standard library contains similar constructed called offsetof, and once again the Palm API provides an alternative which is easy to remember, and simply capitalises the words of the standard function.

Naturally having packed our bug, we also need to unpack it. This is simply a matter of assigning the right value to the variables within the struct. Obviously, if we had more than one string, we would need to use StrLength to figure out the right position for the start of each string in order to assign them. In The code looks like this.

void UnpackBug(BugRecord *bug, const PackedBugRecord	*pBug)
{
	bug->DateEntered = pBug->DateEntered;	
	bug->nBugSeverity = pBug->nBugSeverity;
	bug->nBugType = pBug->nBugType;
	bug->szBugDesc = pBug->szBugDesc;
}

Reading back our records

In this example I have used a GUI component which we have not yet discussed, namely a list. The list has been set to be ownerdrawn, and a callback has been set which will be passed the index of the item to draw. The details of this will be covered in a future article. Our strategy is to use the index passed into the overdrawn function to retrieve the appropriate record from our database and render it. This has the advantage of saving us from making a copy of our entire database, saving both space and processor time. There are two ways to search for a record, by index or by unique ID. The unique ID is not as exciting as it sounds, the search is still sequential and therefore has linear growth in complexity. It's far more common to search by index, but if you have the ID, the API is DmFindRecordByID, and it takes a database, a unique ID and a pointer which returns the index of the item. The DmQueryRecord function takes a database and a record number and returns a record handle or NULL to indicate failure. Here is the full code we use in our list callback to read a record.

Err err = 0;
MemHandle myRecord = DmQueryRecord(gDB, itemNum);

if (!myRecord)
    err = DmGetLastErr();
else 
{
    PackedBugRecord *prec = (PackedBugRecord *) MemHandleLock(myRecord);

	BugRecord * rec = (BugRecord*) MemPtrNew(sizeof(BugRecord));
	
	UnpackBug(rec, prec);

After using WinDrawChars to render our data, we call MemHandleFree to free our records. We do not need to free the record data, because it was never copied out of the database.

Deleting Records

Deleting records is easy, just use DmRemoveRecord(gDB, nIndex), where gDB is your handle to a database, and nIndex is the record to delete.

Some other stuff

Records have a 64k limit, there is a way to overcome this, but I am not familiar with it. Each record also has a record attributes area, which is one byte in size and contains a deleted bit, a secret bit, a dirty bit, a busy bit and a 4 bit category. Records when they are deleted are not necesarily removed from the Palm. DmRemoveRecord deletes them entirely, DmDeleteRecord deletes the record but keeps the header and sets the bit, and DmArchiveRecord does not delete the record, but sets the bit. The reason for these differences is syncronisation, another future topic, which basically involves moving data between a Palm and a desktop application. Deleted records are moved to the end of the database and are not visible, even if they are still there. They are typically always removed at the next syncronisation.

Databases also have their own header, where data like the name, creator, and type is stored, along with some application specific data. This is the place to store data you want to keep only once for the entire database. Additionally, databases can be sorted, through the provided DmInsertionSort and DmQuickSort functions, both of which use a function pointer to allow the programmer to specify the sort order. Insertion while maintaining sort is provided by the DmFindSortPosition API, inserting into an existing record index causes all records below that one to be moved down accordingly.

Although I am unable to comment beyond what the Palm API can tell you about these functions, they seem worthwhile, we have a database at work with 40,000 records and without some sort of binary search, the search times are somewhat untenable.

Conclusion

I hope that this tutorial leaves you feeling able to handle the basics of Palm databases, and aware of some of the more advanced things available to you, all of which are documented in the Palm help. The next topics to cover will be lists and tables, after which we will need to look at conduits and syncronising desktop applications to a Palm. Beginning with the table tutorial, we will be building an actual application, building on the skills covered in these earlier tutorials.