Click here to Skip to main content
15,882,260 members
Please Sign up or sign in to vote.
0.00/5 (No votes)
Hi Team,
I need to convert millions of image to a particular format using third party library. The library provide static functions to convert so I cant use multithreading or parallel.foreach because the library is processing one file at a time.
Second approach what I thought is to create one windows application which will create different directory and copy exe's, dll , configuration file at run time according to the processor available in server and distribute the file to be process count to each exe.

Is there any better way to achieve this ?

Issues : Suppose in case any one of exe hangs, I want to restart that particular instance. How can I do that ?

What I have tried:

I tried to run application using multithreading but third party application throwing exception.
Posted
Updated 4-Aug-23 2:55am
Comments
BillWoodruff 4-Aug-23 11:09am    
what 3rd. party library ? does it provide support ?

converting from/to what formats ?

If the third party is using static methods, then a second Process environment is the simplest (and probably only) alternative - you can do it fairly easily using the Process class to activate an exe and it gets a separate environment even if the DLL is being used already.

The way I'd do it is to write your existing app to process a single file given as a program argument and write a second app that kicks off a number of instances of that program with different files paths passed to them. If you retain the Process object, you can check when it terminates (file processed) and kick off the next one - you don't want to try and run a separate process for a lot of files simultaneously, as it will just slow throughput as each process needs at least one core available in order to run, and if you try to kick of a hundred processes you will slow everything down as you only have 8 cores (probably).

As to detecting the API locking up, you'll have to research the actual API detail; for that - we can't help you!
 
Share this answer
 
Comments
BillWoodruff 4-Aug-23 11:05am    
+5
As @OriginalGriff points out, you can grind your system to a crawl.

You could still use TPL[^] which uses a thread pool. It will manage the load for you based on the number of CPU cores.

To do this, I would write the processing as a plugin DLL (late binding). This allows you to load multiple instances into memory and still communicate between the host app and the plugin DLL instances allowing the host app to display processing information for each running task. You could pool the plugin DLL instances and reuse/recycle to reduce lag from loading and activating new DLL instances.

I did a quick google and found an article here on CP: C# Winforms Plug-ins Architecture Example[^]. I would look at how this works and use it as a guide to building your bespoke TPL processor.

UPDATE

@BillWoodruff questioned if it was possible...

Interestingly, .Net has it's own addin framework that you can use in your app: Add-ins and Extensibility | Microsoft Learn[^]

Here is a better example of what I was explaining: WPF - Build Fault-Tolerant Composite Applications | Microsoft Learn[^]. To quote the article:
Isolation Levels
Microsoft .NET Framework applications can handle third-party plug-ins in at least three different ways:

No isolation: Run host and all plug-ins in a single process with a single AppDomain.
Medium isolation: Load each plug-in in its own AppDomain.
Strong isolation: Load each plug-in in its own process.

It's Strong isolation that I was referring to. See the article for more information on how.

For .Net Core, there is this: Create a .NET Core application with plugins - .NET | Microsoft Learn[^] and also: GitHub - weikio/PluginFramework: Everything is a Plugin in .NET[^] and this: GitHub - merken/Prise: A .NET Plugin Framework.[^] (https://maartenmerken.medium.com/announcing-prise-a-plugin-framework-for-net-core-4af7cf5b4d2b[^]).

Hope this helps.
 
Share this answer
 
v2
Comments
BillWoodruff 4-Aug-23 11:10am    
have you ever implemented such a use case ?
Graeme_Grant 4-Aug-23 11:21am    
I have with MEF and WPF years ago. I did look at doing plugins with WPF like in the article linked & worked fine. As for using multiple instances of the same plugin, no, but the theory is sound. I am curious to try it out myself, but don't have the time atm as I'm deep into a Blazor project which I'll publish in one or more articles. (ps: 1:20am here atm)
BillWoodruff 4-Aug-23 13:02pm    
"multiple instances of the same plugin, no, but the theory is sound" ... famous last words ? MEF now "composition" ... enables maximiainf cpu core use ?



Graeme_Grant 5-Aug-23 20:58pm    
[moved]

If I had time, I would flesh it out with sample code and write an article with the details. It's not something that I will be adding to my to-do list as it is already far too long.

This content, along with any associated source code and files, is licensed under The Code Project Open License (CPOL)



CodeProject, 20 Bay Street, 11th Floor Toronto, Ontario, Canada M5J 2N8 +1 (416) 849-8900