Parallelizing jobs with limited resources

See more:

general

Dear all

I’m trying to parallelize a part of our application but I have some problems….

I need to calculate N jobs. Resources for one job are that big that it is not possible to prepare all N jobs and let them afterwards process by several threads. Therefore I need to feed a limited job queue with M items (where M is pretty much smaller than N)

So what I need to manage is:
1.) There are N Jobs to execute
2.) There is a job queue with maximal M items, feed by a producer thread (?)
3.) There is a thread pool with K worker threads waiting for work

How can I manage/synchronize all these?

- Who will assign/invoke a worker thread its work?
- Should a worker thread -after finishing its job- check the job queue by itself for new work? Or should the worker thread simply go to a “waiting list” and another thread dispatches pending work to worker threads?

Does somebody know a pattern for this?

Thank you very much in advance.
Regards, idle63