An Introduction to Embedded Databases for IoT Developers

A key to any successful application, whether it is processing bank transactions or monitoring a patient’s heart, is processing the data within a specified timeframe. The time it takes to access data creates latency and delays action. Latency is especially a problem with read and write-intensive applications such as sensors and mobile devices using streaming data.

Embedding databases, along with an application on the device, can often reduce latency. Limiting latency is not without cost. We may lose some of the advantages that centralization provides, such as scale and maintenance. Additionally, you can end up with hundreds or thousands of devices, each with a database to maintain, each with a possibility of failure. Therefore, there is a greater need to engineer maintenance and fault tolerance into the solution.

While these embedded databases need to act independently at the device level, it is crucial to recognize that they are not just silos of data. They are part of a network of devices. At least some of the data is transmitted and integrated into higher level databases for further action. Treating these devices as silos either sub-optimizes the value of the data or ignores the complexity of data integration.

There are a variety of embedded databases available. Some key points to look at when evaluating their suitability, in addition to their hardware and software requirements are:

• Do they support Create, Read, Update and Delete (CRUD)?
• Do they meet the Atomicity, Consistency, Isolation, Durability (ACID) standard?
• Are they capable of running on multiple platforms and languages?
• Can they move data to more centralized databases seamlessly?
• Are they zero maintenance and DBA independent?

Ideally, any embedded database would have these capabilities. However, depending on the specific situation, some database solutions choose not to include some of these features in favor of optimizing others.

Database Requirements

In virtually all the situations using embedded databases, multiple complex data streams must be stored in a manner that is suitable for embedded applications to analyze and act. These databases are receiving data from multiple sources and must be capable of performing concurrent read-writes. The database is required to not only store the data, but to maintain its integrity and recoverability in the event of system failure.

While CRUD allows for changing data, it has shortcomings when it comes to maintaining data integrity. ACID provides a higher standard.

A record is written to a database by multiple statements. Atomicity requires the database to accept all the statements that make up the record or none. Consistency prevents invalid changes from being made. Data written in isolation allows writing multiple records concurrently without affecting each other. Durability means that, once a record is committed to the database, it remains committed even in the event of a system failure.

Violating ACID requirements increases the risk of database corruption. When we consider embedded databases, we are not talking about a single database in a single location. In an environment that involves hundreds and possibly thousands of devices, the experience of failure is more likely, raising the importance of having an ACID-compliant database.

Types of Embedded Databases

A variety of different types of databases are available for embedded applications. There are:

• SQL databases such as SQLite
• Time-series databases like Influx
• Data-historian systems

Each of these has characteristics that make them suitable for specific situations.

Traditional SQL databases such as MySQL, SQLite or VistaDB can be used in embedded implementations, but typically require tradeoffs in terms of footprint or features. For example, SQLite implements most of standard SQL. While it is ACID compliant, it uses dynamically and weakly typed SQL syntax. It does not guarantee domain integrity. While databases such as MySQL or Oracle implement a shared repository, SQLite is providing local data to individual applications and devices. It emphasizes simplicity, efficiency, independence, and reliability at the device level. As such, it sacrifices some of its ability to network with other devices and databases.

Time-series databases instrument, observe, learn and automate systems, application and business processes across a variety of use cases. Time-series databases sacrifice some of the characteristics of ACID in the interest of speed. For example, if the system is trying to optimize the number of writes, it may decrease the time that records are locked, affecting consistency. If they are only concerned about the data locally and for a short time, durability may be of limited concern. While these compromises may not be an issue locally, they can create issues in integrating the data at a higher level. Additionally, some time-series databases do not offer full CRUD functionality, prioritizing creating and reading data over updating and deleting it.

Data historian systems are designed to collect and store large quantities of time-stamped data. Generally, they are built to collect readings from numerous devices using a minimal amount of storage with fast retrieval. Data historians are embedded on devices to primarily capture and forward information. While they may provide a SQL-based interface for querying, they do not necessarily follow standard SQL syntax, limiting their usefulness for embedded applications and their ability to integrate with SQL databases without additional ETL development. In general, they are create-read databases and lack many data integrity features.

Actian Zen

The Actian Zen suite of products offers an embedded database that is ACID-compliant and can be integrated seamlessly into a variety of environments from Android and IOS mobile applications to IoT edge devices up to the group and enterprise level, eliminating the need to develop and maintain additional ETL for conversion from one database to another.

Each of the products in the Actian Zen family is Zero DBA maintenance and developer configurable. Each of these factors is important in deploying and maintaining embedded databases across a wide variety of platforms and versions.

Embedded systems need to run across platforms, data management systems and file systems. The Actian Zen family of databases runs on virtually all platforms and supports the seamless transfer of data and most popular programming environments, eliminating the need to develop instances for multiple environments.

Actian Zen installation is hidden in the application (silent install), and the developer can use an API to cherry-pick the components needed and eliminate unneeded features to keep the footprint to a minimum.

Actian Zen Edge Server is designed to reside on Edge and IoT devices. It is a zero-DBA, self-tuning solution that supports SQL and NoSQL access. It has a 30-40MB installed footprint. Edge Server supports the following platforms:

• Linux distributions on Raspbian (Ubuntu, Debian, Arch, etc.)
• Windows 10 IoT Core Server
• Raspberry Pi 2 or higher
• Intel x86/x64 and ARM 32-bit/64-bit

Developers can choose between using full ANSI SQL or API- level access to the key-value store, supported by Btrieve 2 API for native Java, C/C++, and Simplified Wrapper and Interface Generator (SWIG)-generated bindings to scripting languages such as Python, Perl, C#, and PHP.

Actian Core

Actian core is a NoSQL embeddable, zero-DBA, self-tuning database for smart-phones and other IoT devices requiring a small footprint (2MB minimum). It uses the same file format as the rest of the Actian Zen product line, facilitating data portability.

It supports the following platforms:

• Android 5 or higher
• iOS 10.3 or higher
• Android Things 1.0 or higher
• Android SDK 25 or higher
• Xcode 9.3 with iOS SDK 11.3
• Actian Zen Core SDK

It provides direct data access through the B-trieve 2 API. The Btrieve 2 API SDK also includes

SWIG files for Perl, PHP, and Python.

Together with the entire Actian Zen suite of products, Edge Server and Core provide a complete end-to-end solution for database deployment to an IoT environment.

Conclusion

Each type of database solution on the market is optimized to handle a specific type of data or environment. Actian Zen provides the embedded systems developer with an ACID compliant database that is capable of seamlessly networking across a wide variety of environments, speeding up development, and decreasing maintenance costs.