C# Application for Processing on Rows in Different SQL Table

See more:

C#

SQL

Hi,

I am new in the world of SQL database and C# programming, so I am seeking here an advice from experts more than steps or code.

In Brief:

I am making an application that will copy data from a certain database to another one, but the copying will depend on different criteria, this criteria include values in both DBs. Also I want to modify certain values in the newly added rows. I have a couple of ideas in mind on how to do it, but I am concerned about the speed of the process and its efficiency as this copying will be repeated every amount of time (30 secs ).

In Details:

My application will copy data from a Kiwi Created Database (Kiwi is a syslog server that collects Logs from network elements and places them in a DB) to another DB. I want to read the newly added logs from the Kiwi sever and places them in a new DB, if the Logs occurs for the first time, then I add them to the new DB. If they are repeated, some counters should be incremented. Also the Logs usually include source and destination, so I want to consider the logs sent from either the source or the destination as one log. This process should be repeated every certain amount of time (Something around 30 secs).

My ideas about how to do that is:

1. To put both Databases in two array list and then do my processing on them, after that I shall return the values to the DBs.

2. Doesn't use an intermediate array at all, just read row by row from the Kiwi DB and for each row, query the new DB for the values and then update or add them.

Then second method seems to be easier and more straight forward, but it will include too much connections to the DB, I am afraid that will slow down the application.

I hope I represented my question clear enough, I am waiting for your usual support and help.