Kerosene Maps: A dynamic and configuration-less Entity Framework designed for your POCO objects

Introduction

Kerosene is the configuration-less self-adaptive and dynamic ORM library that has been developed to let you avoid all those external mapping and configuration files, dynamically adapting to whatever circumstances it may has to face while, at the same time, letting you use a more natural SQL-like syntax from your C# code. And all of this is possible even if the schema of your database or its details are not known to you, or when they change without your control.

You can find the introductory article of Kerosene in the references section below. I encourage you to read it now before going forward if you have not read it yet. It discuss the basic mode of operation of Kerosene, the non-mapped one, and it is the foundation of this solution.

You may wanto to remember that, in this basic mode of operation, Kerosene operates on "records": instances of the KRecord class. They are dynamic objects able to adapt to whatever schema your database has, without requiring any configuration or mapping files thanks to their self-adaptive capabilities. Kerosene does also provide a converter mechanism to transform those records into instances of your business classes, letting you deal with them in a more abstract and business-oriented way.

While this is enough for a number of scenarios, there are others where having the capability of operate completely at the business entity level is really helpful.

We are going to define such business entities as POCO classes. So they do not have to have any knowledge about database related stuff, and we don't want to require any modifications of their source code, not even in the form of attributes (because it may very well happen that we don't have access to their source code). And, of course, even in this scenario, we don't want to create any wrapper classes (and maintain them if the original ones change), or use any proxy solution of any kind (I like them, but in most scenarios I don't want to introduce more complexity when I'm coding my business solution).

Really, we are going to be serious here: POCO classes must suffice. And Kerosene Maps is built precisely for them.

The Basics

A Kerosene's map is an instance of the KMap<T> class. It takes care of transforming the results obtained from the database into instances of the business entity it is registered for, and to persist them back into the database. Their constructor takes a reference to a link object, and a lambda expression specifying the main/primary table where to find the contents to load the entity. For instance, if we have a class named Region we can create a map for it, in a given link, using the following code:

var map = new KMap<Region>( link, x => x.Regions );

Note that:

• A given business type can be registered only once per each link object.

• To the contrary, as expected, you can have as many different maps as you wish registered in the same link, as far as they refer to different entity types.

• Of course, the same table can be used without restrictions for many maps.

• You can also register the same business entity as many times as you wish as far as each time it is registered into different link instances.

This is all you need to do to work with your business entities as far as the basic Maps scenario suffices for your needs. As soon as you are happy with your map you need to call its Validate() method: Kerosene will try to find a one-to-one correspondence among the columns in your database and your entity’s properties or fields, discarding those columns that don’t match. Kerosene will also find those properties and fields even if they are not public, allowing you to expose only those elements you really want the external world to know.

If a given map is used for database related purposes before its Validate() method has been called an exception will be raised. Once the map is validated you can use it to operate with your entities against the database. On the flip side, if you try to use a map before it is validated for database related purposes, an exception will be raised.

For instance, using the minimalist HR system proposed as an example in the original article of Kerosene, and for the class named Region, you can find a given region using its key column as follows:

Region reg = link.Find<Region>( x => x.Id == "110" );

Kerosene Maps does require that your main/primary table has at least one key or unique column. It is needed because Kerosene will need to uniquely locate the corresponding record when updating or deleting it. But Kerosene does not require you to know what column is the key or unique one, or columns in case you are using a composite key or identity. It will find them out from the database and use them on your behalf as needed.

You may have noticed that, instead of using the map reference, we have used in the example an extension method of the link object. For your convenience Kerosene will locate the map registered for your business type in your link instance and will use it.

The Find() method is somehow special, because it will try to find the entity first in the internal cache (we will see more details about it later). If it is not found here, then it executes a trip to the database to find it. If it doesn't exist in the database, it will return null.

If you rather prefer to query the database to enumerate the results, you can do it easily as follows:

var cmd = link.Query<Region>().Where( x => x.Name >= "E" );
foreach( var reg in cmd ) Console.WriteLine( "\n> Region: {0}", reg );

The Query command accepts the standard methods you may expect, as Where, Top, OrderBy, etc. It does also accept the Join and From methods in case you need to use other tables or sources. In this case, the Query() method can take an optional dynamic lambda expression specifying the alias to use with the primary table, for the sake of this Query command only, in order to avoid name collisions.

Inserting an entity we have created in our code is as easy as follows:

var reg = new Region() { Id = "999", Name = "Solar System" };

var cmd = link.Insert<Region>( reg );
Console.WriteLine( "\n> Command: {0}", cmd );

reg = cmd.Execute();
Console.WriteLine( "\n> Region: {0}", reg );

Note that we have used the Execute() method. The reason is because, contrary to the Find() method above (that is a special case of direct execution for your convenience), all the other Query, Insert, Update and Delete operations do create a command object in order to allow you to get the actual SQL code for logging or debugging purposes.

Let's take a look at how to modify and delete our entities:

reg.Name = "Milky Way";
reg = link.Update<Region>( reg ).Execute();
reg = link.Delete<Region>( reg ).Execute();

Kerosene will find what modifications have been done in your business entity, if any, and will create the update command only for them. What's more, if there are no modifications, the Update's Execute() method won't even try to go to the database.

This is possible because, when using Maps, Kerosene keeps track of your entities and their states. Firstly, each entity carries with it an internal object that maintains the contents of the record obtained from the database, record that is used to find the modifications among many other things.

No, don't be concerned, you have not had not to modify your business entity class to make it happens. You have not had to use any attributes or any proxy mechanism. It is an automatic internal feature. Just for if you are curious: it uses a real-time metadata extension mechanism which closely relates to the one I have developed for Extension Properties. You can find more details in the article that appears in the references section below.

Secondly, it uses an internal cache that also permits Kerosene to avoid duplicate entities. So if, for instance, you obtain the same entity from different parts in your code, their references will point to the same internal object in the cache. If you modify one of those, the other reference will reflect inmediately the changes (they point, indeed, to the same object).

Note that a given entity is internally attached with a given instance of a map once it has been persisted to the database or fetched from it. If you try to use an entity with a map it is not attached to, an exception will be raised.

All the Execute() methods will modify your business entity and return a reference to it, or null if there have been any errors. The obvious exception is the Delete's Execute() method, that will return null in case of success to indicate that the entity deleted is no longer available in the database.

All of this come from free without having to write anything else, not attributes on the classes of your business types, and of course not any external configuration or mapping files.

Going beyond the defaults

If the Kerosene's Maps default mode, as explained above, if enough for your needs, then you don't have to do anything else. Indeed, I have found that many of the entities I end up using can easily fall into this category.

But if you have columns whose names don't match to any property or field in your business entity, if any of those is loaded from many columns or from different tables, or if your business entities have dependency properties (pointing to other parent or child entities), then you want to know how to customize the behavior of your maps so that they are able to perform in this circumstances.

Our example business entity will be the Region one as follows:

public class Region {
   public string Id { get; set; }
   public string Name { get; set; }
   public Region Parent { get; set; }
   public List<Region> Childs { get; private set; }

   public Region() { Childs = new List<Region>(); Clear(); }
   public void Clear() {
      Id = null; Name = null; Parent = null;
      if( Childs != null ) Childs.Clear();
   }
}

It has the "Id" and "Name" properties as expected. But it has a "Parent" property that should be loaded using the contents of the "ParentId" column, and has a list property named "Childs" that we want it to contain the regions that belongs to this one.

Where to specify the customizations

You must place your customizations to the map after its creation but before its Validate() method is called. Once a map is validated it does not accept any more customizations, and any attempt to do it will raise an exception.

But do not forget to call its Validate() method: if you try to use a map that has not been validated an exception will be raised as well.

Specifying what columns to map

Remember that, by default, Kerosene will try to match the columns in your database with corresponding properties or fields in your business class. If no correspondence is found the offending column is discarded. But in our case we don't want the "ParentId" column to be discarded as we will need to use it locate the Parent region. Or in other cases it may very well also happen that you have composite keys, or that you need to use several columns to build the value of a property, or any other scenario you can think.

The very first thing to do in these cases is to tell Kerosene what are the actual columns in the database you want to use in the map you are building, what of those you wish Kerosene to manage them directly, and what of those should not be managed by Kerosene but by your own code instead.

So, if you want to limit the columns retrieved from the database, but you want Kerosene to manage those columns automatically as explained above, you can use the ManagedColumns() method. When you use it, only the columns that you are specifying are going to be used, and become managed columns and properties. It has two overrides: the first one takes a variable list of strings, each of them with the name of a managed column, and the second one is similar but taking a variable list of dynamic lambda specifications.

In our example there is no need to set this delegate, but just for completeness it would look like as follows:

var map = new KMap<Region>( link, x => x.Regions );
map.ManagedColumns( "Id", "Name" );

But the interesting part begins when you want to tell Kerosene that there are columns in the database that you want to use, even if they don’t match with any property or field, or that in any case you don’t want them to be managed automatically by Kerosene for whatever reason. You can use the UnManagedColumns() method as shown in the next example:

map.UnManagedColumns( "ParentId" );

That’s it. The map’s Validate() method will perform some checks and will assure that it has some way with the columns you have specified to uniquely locate the records when needed.

Specifying how to create and clean your entities

When reading from the database Kerosene will need to know how to create instances of your business entities. By default it will try to find a default parameterless contructor in your class definition. If such constructor does not exist, it will raise an exception ... unless you have customized this creation operation. How? By setting the map's CreateInstance property:

map.CreateInstance = () => { return new Region(); };

In our example it is just redundant to set this delegate but, again, I have included the above example just for completeness reasons.

Let me take the chance now to mention some very important and pervasive points:

• The most common way to customize how a map operates is by setting properties like the above's CreateInstance one. These properties are the delegates to invoke when the given operation needs to happen.

• Their default values are null, meaning that a default method will be used instead. If you set them with a not null value, it is going to be the delegate to invoke instead of the default's method.

• Those default methods acts only on the managed columns and properties, so you can use them per your convenience in order not to reinvent the wheel, and concentrate only on your unmanaged columns.

• These default methods have names that start with "Base", as in "BaseCreateInstance()".

For instance, from time to time Kerosene will need to clean a given instance (typically when deleting it from the database, or when some error has happened). If you want to customize the way the cleaning operation happens you can do something like what follows:

map.ClearInstance = obj => {
   map.BaseClearInstance( obj );
   obj.Parent = null;
};

In this example the delegate takes one argument being it the object to clean. Then it invokes the base cleaning method, who takes care of all the managed properties and fields. Finally, the "Parent" property is cleaned explicitly by your code as it is an unmanaged property from Kerosene's automatic perspective.

Specifying how to transfer the information back and forth the database

At its most inner level the Maps mechanism is built on top of the Kerosene’s KRecord one. The reason is that, even when we are using entities, we want to use POCO classes and not to write and maintain any external configuration files. So hence why Kerosene uses the KRecord’s dynamic and self-adaptive capabilities to solve its needs.

These records, instances of the KRecord class, are used by Kerosene as its carry pigeons for many purposes: in particular, to load the instances as the contents are retrieved from the database, and to prepare the record to write back to it when needed. Let's see how can we customize this mechanism for our needs.

When the contents of a given business entity needs to be loaded from the database, Kerosene will, by default, find the properties and fields whose names correspond to columns in the database, and get its contents from the record instance retrieved for this purpose. If you need to customize this operation you can set the LoadRecord delegate, whose first parameter is the transfer record received, and its second parameter is the actual instance of your business type being set (that has been created automatically behind the scenes).

Sounds easy, and it is, but beware because you may think that the following code is correct:

map.LoadRecord = ( record, obj ) => {
   map.BaseLoadRecord( record, obj );
   if( record["ParentId"] != null ) { // This might FAIL !!!
      obj.Parent = map.Find( x => x.Id == (string)record["ParentId"] );
   }
};

But not, it is not, it is just almost correct. What happens is that in some operations not all columns are used, and so the record instance does not have them. So in this scenarios the portion "if( record["ParentId"] != null ) ..." might fail if the specific record has no "ParentId" column.

The record's Schema property can be used to find whether a given column exists or not. But because this need is so common, the KRecord class comes with a handy family of OnGet() methods. Their first argument is the name of the column whose value we want to obtain. Their second argument is the delegate to invoke if, and only if, such column exists. If so, this delegate will take as its first argument the value obtained from the column, letting you do whatever thing you need to do with it. The alternative variant OnGet<T>() let us cast this value to the type "T" before it is used inside the delegate.

So now we are prepared to write the correct code:

map.LoadRecord = ( record, obj ) => {
   map.BaseLoadRecord( record, obj );
   record.OnGet( "ParentId", val => {
      obj.Parent = val == null ? null : map.Find( x => x.Id == val ); } );
};

This code calls the default BaseLoadRecord() method to let Kerosene deal with its managed columns and properties, and then, if and only if the "ParentId" column exists, and if it is not null, uses its value to find the appropriate parent region, and sets the instance's Parent property with it.

It is interesting to note that in this case, to get the parent region, we have used the Find() method. It exists for efficiency purposes and to avoid some dead-lock scenario: I strongly suggest to use it when trying to find an entity inside any of the OnXXX() delegates. It tries firstly to find a cached entity, and if it exists it is returned. If it is not found in the cache, then a trip to the database is executed to load its managed columns.

Let me add an important point:

• The best practice when using LoadRecord is to restrict yourself to those operations that are closely related to the columns being loaded. This is the case with finding the record's parent using the "ParentId" column in the code above.

• But it is not considered a best practice to use this delegate to load other properties that are not directly related to columns in the database. For instance, if you want to load the list property "Childs" this is not the place where to execute a query. We will see later that the OnQuery delegate is the right one.

Let us now look at the opposite direction.

When Kerosene needs to transfer information back to the database it prepares, from your business entity, the appropriate record instance. If you need to customize this process you can set the map's WriteRecord delegate, whose first parameter is the source entity, and its second parameter is the record where to set this information. The next example shows how to do it:

map.WriteRecord = ( obj, record ) => {
   map.BaseWriteRecord( obj, record );
   record.OnSet( "ParentId", () => {
      return obj.Parent == null ? null : obj.Parent.Id; } );
};

The OnSet() family of methods is the one to use to set the values in the record's instance. Their first argument is the column whose value is being set. Their second one is the delegate to call if, and only if, the column exists in the record, meaning that the record is prepared to receive this value. This delegate must return a value that it is going to be the value set in that column.

Querying the database

We are now almost full equipped to execute queries against the database with our complex business types. Your solution will use exactly the same Query method as explained in the basic scenario, but in this case we need to customize the internals of the query operation to deal with the specifics of our business class.

In our case remember we have a "Childs" property that is a list that should maintain the child regions of this region instance. So, we are going to set the map's OnRefresh property with the appropriate delegate, and execute a nested query inside it as shown in the next example:

map.OnRefresh = obj => {
   obj = map.BaseOnRefresh( obj );
   obj.Childs.Clear();
   obj.Childs.AddRange( map.Query().Where( x => x.ParentId == obj.Id ).ToList() );
   return obj;
};

Now you should be used to the pattern: we have called the default method ("BaseOnRefresh()") in order to let Kerosene to deal with the managed columns and properties, and then we provide our custom code. It clears the list, just for safety, and then loads it again executing a nested query.

Here we have just to remember that OnRefresh receives an instance of your business entity, and must also return an instance of your business entity. Whether the returned one is the same instance, maybe modified, as the received one, or a newly created one if you wish so, is completely up to you.

Inserting an entity

To insert an entity there is a similar OnInsert property that you can use to customize how this operation should behave. But before entering into its details, just let me mention a very obvious thing: the map's Insert() method should only be used with new entities, those that you have created in your code. If you try to use it with an entity fetched from the database, or with a deleted one, an exception will be raised.

The OnInsert property, if it is not null, maintains a delegate that receives and should return an instance of your business entity. But if you think twice about the many possible scenarios for this operation, then you wonder how can we face in a proper way the situation when the Parent property is set using a new entity we have just created from our C# code, or an existing one we have modified, or how can we manage the list of child regions adequately.

You always can access to the metadata a given entity is carrying, and based upon its contents and state you can decide what to do. To access an entity's metadata you can use the following code:

var meta = KMetaEntity.GetMeta( obj );

The object this method returns permits you to access the record used used to build the entity (if it was fetched from the database or persisted to it, otherwise is null), what map it is attached to, what is its state, or even what changes it has suffered since it was persisted or retrieved from the database.

I have mentioned the KMetaEntity class here for completeness, but there is an easier way to achieve our objectives: introducing the map's Sync<T>() method. Let's see the actual delegate we want to use for the OnInsert property:

map.OnInsert = obj => {
   obj.Parent = map.Sync( obj.Parent, KMaps.SyncReason.Insert );
   obj = map.BaseOnInsert( obj );
   map.Sync( obj.Childs, KMaps.SyncReason.Insert );
   return obj;
};

You can see we are surrounding the call to the default BaseOnInsert() method with two calls to the map's Sync<T>() one:

• The first one is used to synchronize the instance the "Parent" property may have. If it is null nothing is done. If it is not null, then it takes care on your behalf of inserting, it if it is a new instance, or deleting it and returning null if it has been deleted, or even to update it if needed. It is called before the default method because if it is not null we need to synchronize the record that is, or will be, the parent one of our instance.

• The second call operates on the "Childs" property. As its argument is now a list, it synchronizes all its elements, and updates the list accordingly.
  • If any of those members is null, it is removed from the list.
  • If any of those members is in a Deleted state, it is cleared, and then it is also removed from the list.

In general terms, when using Sync<T>() it takes care of all the complexities on your behalf. You can think in terms of what kind of logic are you using in your entity's properties and forget all the details related to the database. You only need to specify the type as its generic argument (in order to locate the appropriate map), the property you are interested in, either a reference or a list, and the reason of the synchronization using the appropriate enum value as shown in the above example.

Updating an entity

Customizing the way the Update operation works is now very straightforward. It just require you to set the map's OnUpdate delegate with the appropriate code, as for instance th way is shown in the next example:

map.OnUpdate = obj => {
   obj.Parent = map.Sync( obj.Parent, KMaps.SyncReason.Update );
   obj = map.BaseOnUpdate( obj );
   map.Sync( obj.Childs, KMaps.SyncReason.Update );
   return obj;
};

As mentioned above, the map's Update() method will find the changes the entity has suffered since it was retrieved or persisted to the database, and execute an update only for those changes. If there are not changes, this method returns without even trying to reach the database.

Deleting an entity

Let's now take a look at how to delete an entity:

map.OnDelete = obj => {
   map.Sync( obj.Childs, KMaps.SyncReason.Delete );
   obj = map.BaseOnDelete( obj );
   return obj;
};

In this case we don't need to compulsory synchronize the Parent region, just to synchronize (by deleting) its childs as they should not exist any longer once this region is deleted. And it has to be done before deleting our main instance in order to avoid referential integrity errors from our database. OnDelete should return null if the entity has been deleted succesfully.

If you want to later refresh the contents of the Parent region, whose reference is maybe stored elsewhere, you can use the Refresh() method as follows:

theparent = link.Refresh( theparent );

Unlike Find(), Refresh() will try to find the instance in the database without using the cache, refreshing the internal objects accordingly.

What else?

Well, actually there are a lot of things we have not covered in this introductory article. We have only scratched the surface of the metadata extension mechanism and its possibilities. We have not even discussed the "Transient" mechanism Kerosene uses to avoid as many trips as possible to the database, and how this mechanism interacts with the internal cache. We have not discussed the built-in internal collector that in some scenarios can be helpful to avoid the internal cache to grow disproportionately. Etc., etc., etc.

All those discussion are addressed in the article about the internals of the Maps mechanism of Kerosene, as can be find in the references section below.

References

• Kerosene ORM: the original introductory article of the Kerosene ORM library can be found here: Kerosene Basics.

• Easy Extension Properties: describes how to provide support for Extension Properties, the analogous of the built-in C# extension methods, by using an extension mechanism that attach in real-time metadata to your instances. Its details can be found here: EasyExtensionProperties.

• Internals of Kerosene Maps: work in progress.

History

• [v5, September 2012]: Kerosene is the fifth version of this project. It includes the new Maps version built for being an Entity Framework adapted for POCO classes.

• [v4.5, May 2011]: this was a maintenance version that allowed KyneticORM to use any arbitrary type for the parameters of a command, with a transformer mechanism that converts them to instances of objects understandable by the ADO engine. It avoided the need of converting them to strings.

• [v4, January 2011]: this version added support for serialization and WCF scenarios, an improved support for transactions, and corrected some minor bugs.

• [v3, October 2010]: KyneticORM was the third version of the project. It was focused on an improved parsing mechanism and performance.

• [v2, August 2010]: MetaDB was the second version of this project. Its focus was to include some improvements and to resolve some bugs.

• [v1, June 2010]: MetaQuery was the first version of this project. It focus was basically to send queries against a MS-SQL database, with no support of maps, and very primitive Insert, Delete and Update operations.