Ok I came up with solution as:
Delphi side: Created Dll with 6 exports and a little factory inside.
public delegate void ProcDelegate(int value,int callerID);
private static ProcDelegate procDelegate;
[DllImport("OleWrapper.dll", CallingConvention = CallingConvention.StdCall, CharSet = CharSet.Ansi)]
public static extern void InitOle();
public void DelphiCallBack(int value, int callerID)
Init and RegisterCallback called before doing anything else CleanUp destroys the factory.
Only problem is I tried to avoid memory contact with dll so that I used temporary files to pass OLE objects back and forth. Any further suggestion is welcome.