Click here to Skip to main content
Click here to Skip to main content

Movable Global Distributed System with NoSQL

By , 10 Mar 2013
Rate this:
Please Sign up or sign in to vote.

Introduction

These days talking about distributed systems is not just a cool idea, many systems are already running on a group of nodes, but running as a global system is done only by a few big companies. This means thousands of nodes have been placed around the world, with multimillion dollars in costs, paying power and networks charges every hour. We knew how to spend money, I don't need to write an article to emphasize it. In this article, I will discuss a lightweight globally distributed architecture. It is low cost, more importantly, it is movable, which means you don't need to rent a room to place the black, heavy, noisy server.

Background

Think about a small finance company, only having three employees, but they have big ambitions, and run a global business, so they assigned employees in New York, London, and Hong Kong with three notebooks. Modern finances can not run without software systems. This company just started, and employees don't come to office every day, they are more often in airports, highways, coffee houses, beaches, or at a village taking business. Often they don't have a stable internet, but they need the numbers all thetime, so the system must be distributed, and each subsystem must be nearby the employee, better running as local.

Model -Charge free, Globally distributed architecture

First, look at the subsystem, Business Layer, and Database Layer's functionality as its definition. But under the database, we design a Buffer Layer, which is used to mange the changes of the database. If the subsystem is disconnected, it will store the changes to a file, otherwise it will use a faster email system to deliver and receive messages. At the same time, another copy will be sent to the backup address. Each subsystem has a database, so when the system is disconnected, it still works. Registering an email address is free, if your implementation is good enough, maybe you can process thousands of addresses.

Another question you might ask is, how can it be under database programming? We usually work on it. Yes, purely traditional database systems can't do this, but today we have not just SQL, I will use a NoSQL database called "iBoxDB" to make an example.

Message's ID

Since internet appeared, nodes have communicated with each other, then ID design became important. We had many solutions, using datetime, using datatime+delta, using id+serverid etc. Following is a suggested solution, does not always work, depends on the type of the application.

Each node has its own functionality, that is so called "distributed". Each receiver doesn't process all messages, each sender doesn't need to broadcast every message. And because messages might be lost, I introduced a "dependent ID". Iif you want to process this message, you first need to process the message, if lost, send it again.

Demo

Fully implementing this architecture will take a few days, because a globally distributed system needs many double-checks, etc. Fortunately, it is a good design, if you have all, it is great, if only "@Backup.net" is working, it is fine, even if the network is down, then subsystem can run autonomously. In this section, I will only implement the emergency system, "@Backup.net". Because we don't care about IMAP SMTP's details, I just created a fake "@Backup.net" email server, and the messages are just stored in-memory, as a minimized implementation. But you can see just part of the architecture works, the system runs as we want.

This demo includes two commands, "insert [number]" adding a payment record, and "list" showing all records. If input is '1', then start an email server, the code is as follows:

class Program
{
    static void Main(string[] args)
    {
        Console.WriteLine("1 = EmailServer");
        for (var i = 0; i < UserManager.Employees.Length; i++)
        {
            Console.WriteLine(i + 2 + " = " + UserManager.Employees[i]);
        }
        var input = Console.ReadLine().Trim();
        switch (input)
        {
            case "1":
                Console.WriteLine("Email server");
                var server = new FakeEMailServer();
                server.Start();
                while (Console.ReadLine() != "close"){ }
                server.Close();
                break;
            case "2":
            case "3":
            case "4":
                var i = Int16.Parse(input) - 2;
                Console.WriteLine("Login " + UserManager.Employees[i] );
                Console.WriteLine(" commands: 'list' or 'insert [number]' ");
                UserManager.Login( UserManager.Employees[i]  );
                while (true)
                {
                    input = Console.ReadLine().Trim().ToLower();
                    if (input == "close") { break; }
                    if (input == "list") {
                        using (var box = UserManager.GetCurrentDB(UserManager.Employees[i]))
                        {
                            foreach (var x in box.BSelect<Payment>("from Payment"))
                            {
                                Console.WriteLine(x.Employee + "  " + x.Amount);
                            }
                        }
                    }
                    if (input.StartsWith("insert"))
                    {
                        var number = input.Substring(6).Trim();
                        if (number.Length > 0)
                        {
                            decimal num;
                            if (decimal.TryParse(number, out num))
                            {
                                using (var box = UserManager.GetCurrentDB(UserManager.Employees[i]))
                                {
                                    box.Insert("Payment", new Payment()
                                    {
                                        ID = box.NewId(0, 1),
                                        Employee = UserManager.Employees[i],
                                        Amount = num
                                    });
                                    box.Commit().Assert();
                                }
                            }
                        }
                    }
                }
                break;
            default :
                break;
        }
        UserManager.SystemClosed = true;
        Thread.Sleep(1000);
        DistributedDatabaseServer.Instance.Dispose();
        Console.WriteLine("CLOSED");
    }
}

Executing

First, we start an 'email server' and have 'London', 'NewYork', 'HongKong' nodes.

Second, we insert a payment of 88 value in London.net node, you will see the record is distributed to NewYork.net and HongKong.net.

Third, when network is disconnected, so I close the email server and insert 11, 22, 33 in turn, each distributed node working autonomously.

Recover network. the previous records are distributed to each node automatically.

It is amazing! The codes above are so normal. That is the architecture power and the flexibility of iBoxDB. A good architecture can simplify coding, and a nice database lets it further more. The following is the core.

public static void Login(string employee)
{
    (new Thread(() =>
    {
        var db = DistributedDatabaseServer.Instance.GetInstance(GetDBNameFromEmployee(employee));
        var listener = (DistributedDatabaseServer.ChangeListener)db.GetBoxRecycler();
        var changes = listener.Changes;
        while (!UserManager.SystemClosed)
        {
            Socket socket;
            Byte[] change;                   
            while (changes.Peek(out socket, out change) )
            {
                if (socket.DestAddress == BroadCastAddress)
                {
                    bool serverIsClosed = false;
                    foreach (var name in Employees)
                    {
                        if (name != employee)
                        {
                            if (!EMailClient.Send(name, Convert.ToBase64String(change)))
                            {
                                serverIsClosed = true;
                                break;
                            }
                        }
                    }
                    if (!serverIsClosed) { changes.Dequeue(); }
                }
                else
                {
                    changes.Dequeue();
                }                         
            }
            string msg;
            do
            {
                msg = EMailClient.FetchFirst(employee);
                if (msg != null)
                {
                    change = Convert.FromBase64String(msg);
                    change.MasterReplicate(db).Assert();
                }
            } while (msg != null);
            Thread.Sleep(3000);
        }
    }
     )).Start();
}

iBoxDB runs as an embedded database, so you can collect the changes and replicate to other nodes you want, this also means you just add several lines, and a standalone application becomes a distributed application. We use EMailClient to deliver messages, the details are about the email's protocols and these are not shown above. You can download the source and check what happens.

Points of Interest

In the article, I used a global email system as the global message delivery system, because it is free and thousands upon thousands of servers support this huge system. These days, it is hard to hear an email system has crashed. Actually you have many choices, any global service that can post and receive messages is an option, including SMS, Twitter, etc. If you're preparing a big cool system, before paying millions of dollars for servers and networks, maybe you should try this architecture as a homework. After you have bought some things you need to know, if each piece is 99.999% available, when you add up all, the answer is not 1000%, hardware problems are hard to debug, and there is no 100% reliable device, so the functions are not always returning 'true' or 'false', and sometimes are 'undefined'.

References

History

  • V 1.0 (Current).

License

This article, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)

About the Author

Bruce Yang cp
Systems Engineer
China China
Mixing technologies for fun.

Comments and Discussions

 
-- There are no messages in this forum --
| Advertise | Privacy | Mobile
Web01 | 2.8.140421.2 | Last Updated 11 Mar 2013
Article Copyright 2013 by Bruce Yang cp
Everything else Copyright © CodeProject, 1999-2014
Terms of Use
Layout: fixed | fluid