Im trying to process data from my sql server in a chronological manner. The data is too large to fit into memory all at once so i have written a SProc which uses paging and returns X number of records.
To speed up the processing, i create a collection of X number of tasks and assign them different inputs. Each task is assigned different inputs, and they begin read/processing the datatable.
The following is where im stuck:
1)I need a way of pausing each task once it has finished reading all the data from the datatable.
2)Waiting until all the other tasks in the collection have finished reading the datatable.
3)Calling my SProc in order to load the next page of data into the datatable.
4)Recommence the tasks such that their local variable havent changed. ( i was planning to keep the results of each task in a local variable but if this isnt ideal im open to suggestions)