|
I am not sure I will word this correctly as it is currently outside my domain - but I would like some feedback on technologies to look at.
My main app is a GUI and a database engine - once a week data is taken from a large marketing database (58 million records, 400 variables, 39,000 discreet variable values) and compressed and highly indexed into my own proprietary database system - this enables the sales teams to run complex counts and analyses on standard laptops in sub second times.
Currently every monday the sales staff copy the database files to their local laptops (there is no actual data held so no names and addresses etc just bits and offsets)
However the database is now over 6GB and can take 10 minutes to copy on the local network and ages (technical sales term) for those working from home. So I want to write a system where the database is held centrally with a server of some description which my app can talk to specify the information required and receive the results. It could foreseeably have up to 100 users at any one time.
So... my question... what technologies do I need to be looking at? Will WCF suffice?
|
|
|
|
|
RugbyLeague wrote: My main app is a GUI and a database engine - once a week data is taken from a large marketing database (58 million records, 400 variables, 39,000 discreet variable values) and compressed and highly indexed into my own proprietary database system - this enables the sales teams to run complex counts and analyses on standard laptops in sub second times.
What format? Anything you recognize? A zip-file with multiple CSVs?
RugbyLeague wrote: However the database is now over 6GB and can take 10 minutes to copy on the local network and ages (technical sales term) for those working from home.
So they get paid for waiting, getting coffee, and are complaining? Consider having them tested.
RugbyLeague wrote: So... my question... what technologies do I need to be looking at?
See if you can replicate the data on your own system. Then use the Microsoft Sync Framework to synchronize that along the clients. Or, do a binary diff, and sync that by hand, if the users are only reading from the db 
|
|
|
|
|
The database is my own design - essentially a series of bitmapped inverted indexes with run length encoding and data blocks for sparse data - that is all held in one file and there are a couple of supporting small xml files.
I don't want to be copying the data everywhere (as currently happens) - I want the GUI app to communicate to a server app which does the processing on the data held centrally and then communicate back the results.
|
|
|
|
|
You can write a custom plugin for your database, and have the sync framework use it. That'd be more suitable for sharing a large dataset.
RugbyLeague wrote: I don't want to be copying the data everywhere (as currently happens) - I want the GUI app to communicate to a server app which does the processing on the data held centrally and then communicate back the results.
In that case, it'd be better to send it a job (in a single or multiple pieces) and process it on the client. There's multiple ways to communicate the data, you might want to take a deeper look at WCF.
|
|
|
|
General 
News 
Suggestion 
Question 
Bug 
Answer 
Joke 
Praise 
Rant 
Admin 
Submit an article or tip
