Three Useful Programs with Open Storage

Mirzakhmet Syzdykov

3.00/5 (2 votes)

Aug 5, 2022

CPOL

2 min read

13477

Develop programs with open storage using pre-defined format

Introduction

This tip/trick is about the usage of open storage like comma separated values (CSV). We will provide the reader with necessary parsing scheme of CSV files and further usage in utility program composition.

The open storage provides the user with efficient ability to store and parse it just-in-time (JIT).

Open Storage

The CSV format is defined as the line of values separated by either comma, space or any other separated while the string data are enclosed in string holder values. This CSV format can also effectively store real numbers as well and is supported by almost all office programs.

Using the Code

In this section, we will show how to parse CSV file using the simple and efficient approach which is further reused in the applications for every day use: "Alumni" is a stand-alone web server, "CSVdb" is a NoSQL stand-alone CSV database and "ML+" is a stand-alone program to work with Machine Learning and classification.

The CSVLine class parses the line of open storage file on the fly and is realized as follows below:

/*
 * Created by SharpDevelop.
 * User: Mirzakhmet
 * Date: 6/28/2022
 * Time: 1:52 PM
 * 
 * To change this template, use Tools | Options | Coding | Edit Standard Headers.
 */
using System;
using System.Collections.Generic;

namespace Alumni
{
	/// <summary>
	/// Line in CSV file.
	/// </summary>
	public class CSVLine
	{
		public List<string> values = new List<string>();
		
		public CSVLine(ParsingStream stream)
		{
			while (!stream.atEnd()) {
				string result = parseValue(stream);
				
				this.values.Add(result);
				
				if (stream.current == '\n') {
					stream.Read();
					
					break;
				}
			}
		}
		
		public string parseValue(ParsingStream stream) {
			string result = "";
			
			stream.parseBlanks();
			
			int delimeter = -1;
			
			if (stream.current == '"' || stream.current == '\'') {
				delimeter = stream.current;
				
				stream.Read();
			}
			
			if (delimeter == -1) {
				while (!stream.atEnd() && ",;".IndexOf((char) 
                        stream.current) == -1 && stream.current != '\n') {
					if (stream.current == '\\' && delimeter != -1) {
						stream.Read();
						
						result += (char) stream.current;	
					} else {
						result += (char) stream.current;
					}
					
					stream.Read();
				}
			} else {
				while (!stream.atEnd() && stream.current != '\n' && 
                        stream.current != delimeter && ",;".IndexOf((char) 
                        stream.current) == -1) {
					if (stream.current == '\\') {
						stream.Read();
						
						result += (char) stream.current;
					} else {
						result += (char) stream.current;
					}
					
					stream.Read();
				}
				
				if (stream.current == delimeter) {
					stream.Read();
				}
			}
			
			stream.parseBlanks();
			
			if (",; \t".IndexOf((char) stream.current) >= 0) {
				stream.Read();
			}
			
			stream.parseBlanks();
			
			return result.Trim();
		}
	}
}

The bunch of lines is stored in the CSVFile class which is also very quick and robust.

We will provide the code of this class in order to demonstrate the power and effectiveness of open storage based on CSV format.

/*
 * Created by SharpDevelop.
 * User: Mirzakhmet
 * Date: 6/28/2022
 * Time: 1:53 PM
 * 
 * To change this template, use Tools | Options | Coding | Edit Standard Headers.
 */
using System;
using System.Collections;
using System.Collections.Generic;

namespace Alumni
{
	/// <summary>
	/// CSV file.
	/// </summary>
	public class CSVFile
	{
		public string name = null;
		public ArrayList names = new ArrayList();
		public Dictionary<string, int> namesIndex = new Dictionary<string, int>();
		public List<CSVLine> lines = new List<CSVLine>();
		
		public CSVFile(ParsingStream stream, string name)
		{
			this.name = name;
			
			CSVLine namesLine = new CSVLine(stream);
			
			int i = 0;
			
			foreach (string s in namesLine.values) {
				this.names.Add(s);
				
				this.namesIndex.Add(s, i++);
			}
			
			while (!stream.atEnd()) {
				CSVLine line = new CSVLine(stream);
				
				if (line.values.Count > 0) {
					this.lines.Add(line);
				}
			}
			
			stream.stream.Close();
		}
	}
}

Basic Principles

As we have defined the necessary bunch of classes in order to read open storage, now we can construct the configuration files along with other information for our projects which use the open storage like web server "Alumni", NoSQL database "CSVdb" and Machine Learning utility "ML+".

Step by Step Walk-throughs

Thus, we develop the open storage parser for use in the web-server "Alumni", sample files can be found in demo project - these files configure the server:

The NoSQL database which uses the open storage to make queries:

The Machine Learning program to match against the OLAP-like data uses open storage to deal with classified input data within the pre-defined threshold of equality for use queries:

Conclusion and Points of Interest

Thus, we have developed open storage based on the CSV file format for effective usage in data queries and configuration storage. This is not limited to the usage of JSON or HTML and XML formats in databases.

History

4^th August, 2022: Initial version
2^nd November, 2022: Removed links
3^rd November, 2022: Links deleted
4^th November, 2022: License changed