Database

3-Sep-12 22:57

Hello,
I want to store the recent measuring results as well as control samples of older measurements persitently on disk so that the last state can be restored even in the case of a power loss. To make loading the data on startup not unnecessary slow, I want to remove the old, unneeded data from the list.

This means that the measuring results will be continuously appended to the list, but the time when it will be removed is not known in advance.

Would you recommend using a database (ideally some compact system like SQLite so that the user does not have to install a databse server) for this, or are there alternatives?

My idea would be to store the measuring results in one table, using the measuring time as key. In another table, the keys for the last n measuring results would be stored and in a third table the keys of the control samples. If the entries of the first table are not referenced any more from the two other tables, the measuring results should be automatically removed. Is there some concept to acchieve this?

Alex

Mycroft Holmes4-Sep-12 0:40

Mycroft Holmes

4-Sep-12 0:40

This decision should be driven by volume and usage. If you are only processing a few hundred records and do not need to massage the data then an XML or csv file store would be acceptable. Otherwise go for a database.

Why make it complex with multiple tables if you are using text/xml, simply load the entire file into memory and manipulate the collection. Write the collection to the file at regular intervals.

A lot will depend on the criticallity of the data and the frequency of the reads!

Never underestimate the power of human stupidity
RAH

LionAM4-Sep-12 4:37

4-Sep-12 4:37

Thank you for your answer.
In a previous version, we wrote the complete XML file - however, due to the relative large amount of data (and the limited resources, as it is an embedded system) - this will not be possible in the final release. So I think the database is the most promising.

Which leaves me with the other questions, how to implement this as a weak reference pattern.

Alex

Shameel4-Sep-12 5:50

Shameel

4-Sep-12 5:50

LionAM wrote:
how to implement this as a weak reference pattern.

Databases generally do not directly support the equivalent of weak reference pattern.

One solution is to append the data as needed in a database table and then run a program in background periodically and check for "old" data and purge them from the database. What "old" means and the frequency of purging depends on the volume of your data.

PIEBALDconsult4-Sep-12 5:56

PIEBALDconsult

4-Sep-12 5:56

As a database kind of guy, I strongly recommend a database.

LionAM wrote:
To make loading the data on startup not unnecessary slow

I asssume you do this to show a graph or similar of the recent trend, otherwise I don't see why you would load old data. A database makes this easy by allowing you to query only the recent data. You could sort the data descending by date and take only the top n records or you could query the records where the timestamp is greater than the current time minus soome number of minutes or hours. Or you could purge data out of the table after some period (perhaps after summarizing it into another for historical reasons).

Also, as I don't know what sorts of "measuring results" you are dealing with I'll pass along a concept that you may find useful. It's sort of a data compression technique whereby if the measurements don't change much you don't store every individual measurement. For instance, the temperature of a commercial oven for baking bread or cookies needs to be monitored, it can probably vary a little (let's say a tenth of a degree) from the target temperature without causing an alarm, but if it changes a whole degree (for instance) it should be noted. Furthermore, if the temperature stays close to the target temperature for hours on end (as it should) we do want to log some readings (perhaps at least every fifteen minutes). For these reasons, you may want to develop a system that will only store a reading if the difference from the previous stored reading is outside some specified range or if some specified time span has elapsed.

LionAM4-Sep-12 10:51

4-Sep-12 10:51

In principle, this is exactly what I want - store a live view of a certain time span as well as the status of several control charts. As the statistical spread of the measurement results is of great interest, it makes no sense to store only large fluctuations (then it would perhaps be too late to countersteer ...).

As the data is only used to restore the last program state and therefore stored locally (there is a separate export feature), I would prefer some compact database (without a database server). Which would you recommend? SQLite?

Alex

PIEBALDconsult4-Sep-12 11:58

PIEBALDconsult

4-Sep-12 11:58

LionAM wrote:
it makes no sense to store only large fluctuations

You can set the thresholds as appropriate.

LionAM wrote:
some compact database

I've dabbled in Sql Server Compact Edition recently, but I can't really give an informed recommendation.

You could also consider serializing the current state to a file periodically.

Maybe my LimitedQueue[^] could be of use to you.

Eddy Vluggen4-Sep-12 21:43

Eddy Vluggen

4-Sep-12 21:43

LionAM wrote:
SQLite?

I'd recommend programming against the IDbInterfaces, and actually time/test the various databases that you would want to consider.

Sqlite is good in terms of performance, once you take in account that everything is in a transaction.
SqlCe is more integrated into .NET, and would be the preferred option if you plan to synchronize to Sql Server.
Microsoft Access would be a sweet idea if your client has Office, and if the primary goal of the data is to show reports.
Excel would work too, almost as fast as Access, and can be accessed over ADO.NET; Not an obvious choice as a datastore, but still very nice for charts.

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

Jörgen Andersson5-Sep-12 2:07

Jörgen Andersson

5-Sep-12 2:07

Eddy Vluggen wrote:
xcel would work too, almost as fast as Access, and can be accessed over ADO.NET;
Not an obvious choice as a datastore, but still very nice for charts.

And if the primary goal of the data is to be accessed by accountants. Smile | :)

Light moves faster than sound. That is why some people appear bright, until you hear them speak.
List of common misconceptions

LionAM5-Sep-12 20:40

5-Sep-12 20:40

Thank you for your answer.
The data is used only to restore the last program state. There is an additional export feature - so I think SQLite ore SQL CE will achieve what I want.

Alex

Connect to MySQL server

1.fcb191331-Aug-12 0:07

1.fcb1913

31-Aug-12 0:07

Hallo,
I cannot connect to MySQL database from PC outside home network.
I created client aplication in c++ to editing data in MySQL database.
We have 2 PC at home. Both PC has the same TCP/IP address like xxx.xx.xx.xx.
PC with MySQL server has IPv4 address 10.10.11.93. I tried to type xxx.xx.xx.xx/10.10.11.93 but
error is occurred: 2005: unknown MySQL server host "xxx.xx.xx.xx/10.10.11.93" ( 11004).
I can connect to the database from second PC at home wrighting the name of PC with MySQL server.
What I have to write to connect from outside?
Thanks.

Eddy Vluggen31-Aug-12 1:26

Eddy Vluggen

31-Aug-12 1:26

1.fcb1913 wrote:
What I have to write to connect from outside?

In a default-installation, it's not possible to make a connection to the server from the outside world. The idea behind that being that you can install a server, without having to worry about me poking at the password over the internet.

You'll have to check the local firewall, and, if it's a home-PC, your router. If that last is the case, you can go to this site[^], and click on "More about you" on the left; it'll load a new page, hopefully with a hostname. If not, you can still connect using the IP-adress, given that the server is configured and reachable.

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

1.fcb19131-Sep-12 22:50

1.fcb1913

1-Sep-12 22:50

On MySQL server is enabled TCP/IP networking, firewall exception for port and enabled root access from remoute machines. I know my TCP/IP - is static.
If I'm trying to connect from outside home network, a firewall don't notice anything.
I can connect from another PC at home network. What I have to do to connect from outside?

Eddy Vluggen1-Sep-12 22:57

Eddy Vluggen

1-Sep-12 22:57

1.fcb1913 wrote:
What I have to do to connect from outside?

I suggest you run a little webserver, and use that to verify whether you can connect. If you can't serve a page from the machine, then you won't be able to use a database remotely.

The logical steps would be something like
- verify connectivity
- verify db-server is running
- verify local connectivity
- verify remote connectivity

If your PC is "at home", chances are your router has it's own firewall. Check the manual to verify - the router-setup varies per product. I changed something called "NAT", pointing from the outside to the static IP on the inside-network.

(The numbers inside the network should be different from those outside, and you'd be using the outside-number to connect)

Bastard Programmer from Hell Suspicious | :suss:

if you can't read my code, try converting it here[^]

jschell2-Sep-12 7:51

jschell

2-Sep-12 7:51

1.fcb1913 wrote:
server has IPv4 address 10.10.11.93. I

That is a private IP address.

It is NOT possible to address a private IP address on the internet. There is absolutely no way to do it.

Your options are
- Figure out your public IP address, which your ISP has given you. There is NO guarantee that you have a public IP address. However if you have one then your cable modem would be the one that has it. You would need to set up your modem to redirect requests to your box.
- Buy a public IP from your ISP. Then do the above.

1.fcb19134-Sep-12 6:59

1.fcb1913

4-Sep-12 6:59

Thanks.
I have public IP address.
On a chimney we have white cubic transmitter-receiver and one cable ended with a switch from witch the computers are connected.
I don't know if it is router or modem.
I tried typed my public IP address, IPv4 address, gate IPv4 and server DNS IPv4 address into web browser to set up connection,
but always connect error is occured. I have contacted my internet provider, I hope he'll resolve this.

Paul Conrad4-Sep-12 7:14

Paul Conrad

4-Sep-12 7:14

Have you tried using port forwarding as Bernhard has suggested?

"Any sort of work in VB6 is bound to provide several WTF moments." - Christian Graus

Bernhard Hiller2-Sep-12 21:09

Bernhard Hiller

2-Sep-12 21:09

Depending on the hardware for connecting your local intranet to the internet, it may be possible to set-up a "port forwarding": that means, you send the request to your public ip address with a port specified, and the router then forwards the request to an internal computer and that port. To get your public ip address, you can use tools like dyndns.