Distributed computing in small and middle offices

Introduction

There are several tasks in game development tools programming, which require large computational power. For example, it takes up to 24 hours to compute game level lighting in our level editor.

Obviously – the longer is operation, the less frequently it will be used, and we will get results with less quality, because a lot iterations are required for good tweaking.

Recently, I gave a lot of attention to GPGPU – using GPU power to accelerate tools. Early version of our level editor did lighting calculation (Ambient occlusion[2] + direct lighting) by rendering hemicube side views from every texel of lightmap.

Unfortunately, experiments show, that simple CPU-only raytracer can do this task 2-3 times faster, because of slow Videomemory<->CPU transfer rate. Possibly, I could have achieved better results, if whole algorithm is implemented on GPU. But this would have require a lot of work due to complex GPGPU programming model. Only recently, nVidia have provided relatively easy C framework for GPGPU programming[3][28].

Multicore CPUs have shown good perspective it this area. The second core, unlike GPU, can execute the same binary code. There is no need to adapt algorithms for different programming model and language, to take advantage of additional performance.

Before I even started to optimize tools for multicore CPUs, I understood that I need to think wider :)


Before we continue, I will ask reader to press CTRL+ALT+DEL and switch to “performance” tab. Just look at “CPU Usage” graph. I bet most readers will see value less then 10% there. On dualcore CPUs, second core will be free most of the time.

During typical work, full computer power is used only in the short moments of generation of response to the action of user. When user is typing text in Microsoft Word, pauses between keyboard events are comparable to eternity from processor’s view. Indeed, number of applications, fully utilizing processing power, are not very high.

Now count up, what power is available in middle office (50+ workstations) – and you will get value, comparable to modern supercomputers.


Unfortunately, traditional relation of cluster calculations to high science, long time served as a mental barrier for me to attempt to use distributed computing in everyday tasks. Fortunately, excellent application – IncredyBuild[4] has shown that distributed computing can be used not only for everyday, but also, possibly, real-time tasks.

Data-parallel programming model

A number of workstations, connected by TCP/IP network, can be considered as supercomputer with distributed memory architecture. The most simple pattern for these systems is data-parallel computing.



In other words, if some ordinary application processes array of independent elements, than in data-parallel model, each processor is assigned to process some part of array.

To support data-parallel computing, core library should divide tasks into parts, transfer task data to local memory of particular CPU, run the task on particular CPU, transfer back results to caller, and provide ability to request some global data from caller.

This work is done by cluster support software. I have searched for available software, taking in account the following requirements:

• workstations are running Windows XP;
• only idle time is allowed for usage by cluster. Workstation user should not experience any slowdowns;
• high tolerance – cluster is not dedicated. Any workstation can be rebooted, go offline or get busy by user, and this should not affect overall cluster stability and state.

The first requirement put aside most of the candidates[5][6]. The reason is clear – why would someone buy commercial OS for every cluster workstation, if there is freeware linux?

Second and third requirements put aside all other candidates, although I would like to mention some of the most promising libraries.



MPICH2[7]. Windows implementation of MPI library. Unfortunately, it runs hidden processes with normal priority, which affects workstation user. If some workstation goes offline – MPI session terminates with error.

libGlass [8]. Despite easy programming model, I have put aside this candidate, because there were no new versions of this library for a long time.

Alchemy[9]. The most promising candidate. Provides easy to use programming model, works on .NET framework, which, buy the way, in theory can allow even Windows Mobile smartphone, like Motorola MPX 200, to became a node in the cluster :). Contains wonderful examples and continues to evolve.

Unfortunately, during my testing, some examples crashed and left orphan processes on nodes.

After unsuccessful search, I made a decision to develop library myself.

hxGrid

During development, I was taking into consideration the following additional requirements:

• TPC/IP 10-1000 MBit local network is used for data transfer;
• a network is considered internal and safe; library does not contain any authentication protocols. Good security system would increase development time dramatically, and make library usage more complex;
• “time to send task” / “time to complete task” ratio is 0.1 and more. Typical task data size can be 0.001-100Mb;
• there is some workstation in local network, which is always online and can run coordinator(manager) application;
• library will be used both for offline (up to several hours) and semi-real-time (up to several seconds) tasks;
• single workstation can run only one grid application at given period of time (this is limitation of current version);
• for debugging purposes, single workstation should be able to run agent, coordinator and grid application at the same time;
• when some workstation goes online, it should be able to join already running grid sessions;
• when some workstation goes offline, this should not break overal process. hxGrid just sends incomlete task to other node;
• cluster can be used from several workstations at once; computational resources should be redistributed equally;
• grid agent should be able to use all CPU cores on each workstation;
• library should be available for use in C++ and Delphi.

To minimize development time, I have chosen to implement library on Delphi. Library is using fake COM interfaces [11]. Library is using Jedy Visual Code Library[15] and ZLIB[27].

Cluster installation

Binary package should be downloaded to install cluster[10].

The cluster software consists of three components:

1. Coordinator. Coordinator should be installed on workstation, which is always online. Coordinator is responsible for maintaining a list of computational resources, and distribution of this list to grid users. Coordinator should be installed on only one workstation in local network.

Agents and Users can find Coordinator by sending broadcast packets in local network. Thus no configuration is required if Coordinator is moved to PC with other IP address.

Users connect to Coordinator only to get list of agents. Later, users connect to agents directly, unlike other grid solutions, where coordinator distributes tasks among agents. . This allows to establish fastest communication between user and agents.

2. Agents should be installed on every workstation in local network. Agent allows utilizing idle CPU time of workstation.

3. Grid user – some application, written with hxGrid library.

Programming for hxGrid

hxGrid allows to run tasks (execute procedures) on cluster workstations. Grid application places tasks into queue and waits for completion.

Input data for tasks are written into stream (IGenericStream inStream). After successful execution of task, output data is received also in stream (IGenericStream outStream).

If necessary, an agent, executing a task, can request additional data from grid application.

Every task is function with the following signature:

C++

typedef bool (__cdecl TTaskProc)(IAgent* agent, DWORD sessionId, 
                                 IGenericStream* inStream, IGenericStream* outStream);

Delphi:

type TTaskProc = function(agent: IAgent; sessionId: DWORD; 
                          inStream: IGenericStream; outStream: IGenericStream): boolean; cdecl; 

where sessionId – unique session id. This Id is required to request additional data from grid application (described later). Function should return false, if interrupted (described later).

Such procedures should be placed into DLL. hxGrid delivers the code by transferring DLL to other workstation. TTaskProc function should be thread-safe.

After successful execution of task on agent, library sends output stream back to user, and calls FinalizeTask()callback:

C++:

typedef void (__cdecl TFinalizeProc)(IGenericStream* outStream); 

Delphi:

type TFinalizeProc = procedure(outStream: IGenericStream); cdecl;

This function should be thread-safe.

If single threaded application is processing large array of independent elements, the obvious way to increase performance is to divide array into small ranges and process them on cluster nodes.

How it works

When started, each agent looks for coordinator, sending broadcast packets over LAN. When connection with coordinator has been established, agent starts sending self status periodically (percentage of free CPU time, amount of free physical memory, number of queued tasks).

To begin cluster usage, grid application initializes hxGrid library. Library connects to coordinator (sending broadcast packets, if required), requests list of agents and connects to them directly. From this moment, library is ready to execute remote tasks. At the same time, library continues to request list of agents, so new agents are able to join running session.

Grid application adds tasks to queue. Library continuously sends tasks to agents in background thread. If connection with some agent is lost, library is able to reassign it’s tasks to other agent.

When library is deinitialized, it disconnects from the agents.

To increase efficiency, the following technical decisions were applied:

• to compensate time of network communication, library sends next task to each agent ahead;
• in some circumstances, library can send copy of task, already running on some agent, to other free agents. This can help to complete job faster, if agent is slow or have no free CPU time at present moment;
• library tracks number of free CPU time and amount of free physical memory on agents. Library will not send task to agent, if it does not have free CPU time (percentage is specified in settings), or have less free physical memory, then required to store input stream (factor is configured in settings);
• library controls length of task queue and size of input streams. If overall size of queued input streams exceeds configured value, library blocks application execution until at least one task is completed;
• if special option is selected, library can swap task queue to disk, to minimize memory usage;
• before sending large stream over network, it is compressed with ZLIB (size threshold is specified in settings);
• agent is caching DLLs locally during the session; DLLs are not resent to execute next same task;
• if agent does not have free CPU time for 10 seconds, this means that user has launched resource-hungry application. In this case, agent suspends all working threads for 25 seconds. This is necessary, because otherwise Windows scheduler will dedicate 120ms of CPU time to working threads with IDLE priority, when CPU starvation is detected[12]. If threads would not have been suspended, this could lead to jerky FPS in games, for example.

Library usage

Library is distributed in form of DLLs: hxGridUserDLL.dll, zlib.dll.

To start session, grid application should create IGridUser object (see examples\picalculator\C++ и examples\picalculator\Deplhi):

C++:

#include "hxGridInterface.h"

void main()
{
 IGridUser* user = CreateGridUserObject(IGridUser::VERSION);
 ...

Delphi:

Uses T_GenericStream, I_GridUser;

var
 user: IGridUser;

begin
 IGridUser_Create(user);
 ...

From this moment, it is possible to execute tasks on cluster nodes:

C++:

IGenericStream* stream = CreateGenericStream();
DWORD d=1+i*9;
stream->Write(&d,4);
user->RunTask("picalculator_task.dll","RunTask",stream,Finalize,&d,true);

Delphi:

//write input data to stream
stream := TGenericStream.Create();
d := 1+i*9;
stream.write(d,4); 
//add task to queue
user.RunTask('picalculator_task.dll','RunTask',stream,Finalize,d,true);
//now library owns stream, release our ref 
pointer(stream):=nil; 

RunTask() method has the following parameters:

• dll filename with function code;
• symbolic name of function (function should be exported from DLL by name);
• input stream. Library is receiving ownership of the stream object;
• completion callback address;
• address of variable to receive unique id of task;
• blocking flag.

If task can not be added to queue immediately (due to limitations of queue length or queue input streams size), and blocking flag is set, method will not return until task is added to queue. Otherwise method will return S_FALSE, and application can wait for good moment with User->WaitForCompletionEvent()).

Please note, that some dependent DLLs, for example, VC++ runtime libraries, can be missing on remote workstation. It is advised to build with static libraries, and always check dependencies with dumpbin utility. To send additional DLLs to workstation, their names should be specified in RunTask() method, separated with comma:

user.RunTask('GridGMP_task.dll,GMPPort.dll','RunTask',stream,Finalize,d,true);

Task function should call the following agent method periodically, to allow agent handle abortion and suspension events:

Delphi:

if (agent.TestConnection(sessionId)<>S_OK) then
 begin
  result := false;
  exit;
 end;

This is very fast method, and can be called from inner cycle even with ~10000Hz frequency.

User->WaitForCompletion() is used to wait until all queued tasks are done.

To close session, application should destroy IGridUser object:

C++:

user->Release(); 

Delphi:

user:=nil;

This small chapter has shown everything needed to use hxGrid.

Requesting additional data from user

In some circumstances, tasks, running on agents, should access some global data, common to all tasks in session.

If global data size is high, it is inefficient to pass global data in task input stream. Agent provides the following method to request global data from user:

C++:

virtual HRESULT __stdcall GetData(DWORD sessionId, const char* dataDesc, 
                                  IGenericStream** stream) = 0;

Delphi:

function GetData(sessionId: DWORD; dataDesc: pchar; 
                 var stream:IGenericStream): HRESULT; stdcall;

dataDesc – symbolic identifier of data, f.e. ‘geometry’;

stream – address of variable to receive address of stream with data (stream is created by agent, and should be freed by task).

To handle requests, hxgrid application should register the following callback:

C++:

typedef void (__cdecl TGetDataProc)(const char* dataDesc, IGenericStream** outStream);

user->BindGetDataCallback(GetDataCallback);

Delphi:

type TGetDataProc = procedure(dataDesc: pchar; var stream: IGenericStream); cdecl;

User->BindGetDataCallback(callback: TGetDataProc); stdcall;

TGetDataproc should create stream and fill it with data. Ownership of stream is passed to library.

Usually, task is trying to access some global cache, and, if data is not in there, requests data from hxgrid application. Cache should be indexed with sessionId and data symbolic description.

This methods are used in examples\normalmapper-3-2-2 to request hi-poly mesh octree.

Additional methods

C++:

virtual void __stdcall GetSettings(TGridUserSettings settings);
virtual void __stdcall SetSettings(TGridUserSettings settings);

Delphi:

User.GetSettings(var settings: TGridUserSettings); stdcall;
User.SetSettings(var settings: TGridUserSettings); stdcall;

Methods allow to change settings programmatically. For example, application can disable stream compression, if it fills streams with compressed data.

C++:

virtual HRESULT __stdcall CompressStream(IGenericStream* stream);

Delphi:

User.CompressStream(stream: IGenericStream): HRESULT; stdcall;

This method allows to compress stream. For example, it is good practice to prepare and compress stream with global data ahead, so library will not have to compress stream each time GetDataCallback() is called. Library is able to find out if stream is compressed, by examining signature. Compressed stream should not be unpacked by task – agent does it automatically. See examples\normalmapeer-3-2-2.

C++:

virtual HRESULT __stdcall FreeCachedData(DWORD sessionId, const char* dataDesc); 

Delphi:

HRESULT __stdcall IAgent::FreeCachedData(DWORD sessionId, const char* dataDesc) = 0;

Agent is caching global data, requested during session. This method allows to free cache and minimize memory usage, if it is known, that data will not be requested again during session.

For example: task is requesting global data, builds some structures from it, and stores them in some custom global cache. Other tasks will use structures from this cache and will not call agent.GetData() during session anymore.

C++:

typedef void (__cdecl EndSession)(IAgent* agent, DWORD sessionId);

Delphi:

type TEndSessionProc = procedure(agent: IAgent; sessionId: DWORD); cdecl;

Agent calls EndSession() callback at session end. Usually this callback is used to free global session data cache.

Callback should be exported by name from DLL, which was specified to IGrudUser->RunTask(). This callback is optional.

Cancellation

There are two patterns to use library:

1) tasks are added to queue with IGridUser->RunTask(blocking = true). Then, application waits for completion with IGridUser->WaitForCompletion().

2) Tasks are added to queue with IGridUser->RunTask(blocking = false). If method returns S_FALSE, because of queue is full, application waits for queue progress with IGridUser->WaitForCompletionEvent(). When all tasks are submitted, application waits for tasks completion, calling IGridUser->IsComplete() periodically.

Delphi:

for s:=1 to 200 do
  begin 
   ... 
   while (user.RunTask('GridGMP_task.dll,GMPPort.dll',
                      'RunTask',stream,Finalize,d,false)<>S_OK) do 
     begin 
       Application.ProcessMessages();
       user.WaitForCompletionEvent(100);
       if state=ST_CANCELING then 
         begin 
           break; 
         end; 
     end; 
   if state=ST_CANCELING then break;
   ... 
  end; 
... 
bl:=true;
while (bl) do
  begin 
    Application.ProcessMessages();
    Sleep(100); 
    if (state=ST_CANCELING) then
      begin 
        user.CancelTasks();
        break; 
      end; 
    user.IsComplete(bl);
  end; 

Application can cancel all tasks at any moment by calling IGridUser->CancelTasks().

Despite bigger complexity, second pattern has the following advantages: application can perform some tasks while waiting; there are ability to cancel tasks and ability to show progress in UI.

For details of second pattern, see examples\GridGMP\.

Examples of library usage

PI Calculator (examples\PICalculator)

The most simple example. This is Pi value calculation with specified accuracy. Each task is calculating [n…n+8] number.

Mandelbrot (examples\mandelbrot\)

Mandelbrot fractal explorer. There are three versions of application:



SingleCPUExtended – this example is using FPU native 10-byte floating point type and runs on single CPU. Luck of accuracy does not allow high zoom levels.

SingleGMP – this example is using 256-byte «big numbers» on single CPU. A Delphi port of GMP[13] library is used.

GridGMP – the example is using 256-byte «big numbers» and hxGrid. Each agent is assigned to calculate one vertical line of the image. This application also shows how badly partitioned calculations can decrease overall performance: vertical lines can require very different amount of processing power, up to 50:1. A better solution would be to assign each agent to calculate every N’s pixels of image’s horizontal scanning.

Normalmapper (examples\normalmapper-3-2-2\)

ATI NormalMapper 3.2.2[14], ported to hxgrid.

Normalmapper is ideal application for distributed computing. Each texel of normal map can be calculated independently from others. Total time to calculate normal map with occlusion term and bent normals enabled, can take up to 12 hours.

The modified version forms tasks to calculate 100 pixels of normal map. Each agent requests serialized hi-poly mesh octree by calling IAgent->GetData(). Octree size can be up to 300Mb, so sending it in task input stream is inefficient.

In addition, modified version contains mutex, which allows to run only one instance of normalmapper at a time. It is possible to run several instances with different models, and they will run in queue automatically.

It should be mentioned, that library does not warranty the order of tasks completion. Since it is important to calculate texels in strict order (because of intersecting mapping and border texels), applications performs special procedures to update normal map texels in required order.

How to install ATI NormalMapper for hxGrid

System requirements:

A couple of PCs running Windows 2000/XP SP1/SP2/Vista, connected via local network.

1. Download hxGrid Coordinator[10] and install on any workstation in the local network. Coordinator should be installed only on one workstation in the local network. This PC should always be available to coordinate hxGrid.

2. Download hxGrid Agent[10] and install on all workstations in the local network.The more PCs are running the agent, the more faster will hxGrid apps work.

3. Download ATI Normalmapper for hxGrid[10] and install on any PC in the local network.

No additional configuration is required.

Hint: If everything above are installed on single workstation, and it does not have network adapter with IP address, "Microsoft loopback adapter" can be installed and assigned some IP address.

Results

Workstations configurations:

1. Intel Pentium D Presler 3.0GHz
2. Intel Pentium 4 2.8 GHz (HT)
3. Intel Core 2 Duo 1.8GHz
4. Intel Core 2 Duo 2.13GHz
5. Athlon X2 3600+
6. LG GE notebook (Intel Celeron M 1.4 GHz)

Measurements were made for calculation of car_low.nmf+car_high.nmf meshes, 4096x4096, Bent normals, Ambient occlusion.

Normalmapper.exe –on carlow.nmf carhigh.nmf 4096 4096 test.tga

For experiments, original version of ATI Normalmapper has been recompiled with VS 2005 with aggressive optimizations (no PGO).

Experiments show, that hxGrid can be successfully used to take advantage of multicore CPUs even on single workstation (agent, coordinator and grid application is running on single workstation).

In other worlds, calculation of normalmap on cluster took 8 minutes, comparable to almost 4 hours on single PC (performance improvement – 27,4 times).

HT CPU showed expected 12% speed improvement, but I can’t explain 35% speed up of Core 2 Duo (I expected ~80%). Probably, the bottleneck here is shared L2 cache, since Pentium D Presler showed 73% speedup.

State of cluster (end of working day)

Energy consumption

Because of high energy consumption, cluster room should be equipped with ventilation and cooling. How much does energy consumption of office increase while agents are active?

100% processor load increases energy consumption approximately by 30 Watt. Accuracy of my measurements can be questionable, but even on 50 workstations office overall energy consumption increases only for 1.5kWt, which is less then good teapot (1.8kWt).

Known problems

The following problems have not been solved yet:

• disconnection from grid could take up to 30 seconds, due to impossibility to close connection on server side ("features" of TServerSocket Delphi component). This is not a problem when using Distributed Normalmapper, but can become a problem when hxGrid is used to solve semi-realtime tasks. If someone can help me with solution – I would appreciate this[29].
• StartSession()/EndSession() callbacks are not yet implemented;
• There are no easy error handling and debugging support.

Public applications using hxGrid

Currently, xNormal[29] is using hxGrid to accelerate normalmap generation.

Links

1. General-Purpose Computation Using Graphics Hardware

http://www.gpgpu.org/

2. Ambient occlusion - From Wikipedia, the free encyclopedia

http://en.wikipedia.org/wiki/Ambient_occlusion

3. NVIDIA CUDA Homepage

http://developer.nvidia.com/object/cuda.html

4. Incredibuild by Xoreax software

http://www.xoreax.com/main.htm

5. Globus Toolkit Homepage

http://www.globus.org/toolkit/

6. Кластерная система Condor

http://www.osp.ru/os/2000/07-08/178077/

7. MPICH2

http://www-unix.mcs.anl.gov/mpi/mpich/

8. libGlass – distributed computing library

http://libglass.sourceforge.net/download.php

9. Alchemy – distributed computing library

http://www.alchemi.net/index.html

10. hxGrid binaries

http://sourceforge.net/project/showfiles.php?group_id=236427

11. Программирование с использованием COM-подобных интерфейсов.

http://www.dtf.ru/articles/read.php?id=44995

12. То, что вам никто не говорил о многозадачности в Windows

http://www.dtf.ru/articles/read.php?id=39888

13. The GNU MP Bignum Library

http://gmplib.org/

14. ATI Normalmapper

http://ati.amd.com/developer/tools.html

15. Jedy Visual Code Library

http://homepages.borland.com/jedi/jvcl/

16. PlayStation 2: Computational Cluster

http://arrakis.ncsa.uiuc.edu/ps2/cluster.php

17. Unmodified Xbox Cluster

http://www2.cs.uh.edu/~bguillot/xbox/home.html

18. Информационно-аналитический центр parallel.ru

http://www.parallel.ru/

19. Распределенные вычисления: поиск лекарства от рака

http://www.3dnews.ru/reviews/software/cure-for-cancer/

20. РАСПРЕДЕЛЕННЫЕ МОЗГИ

http://www.fuga.ru/articles/2003/01/distributed.htm

21. Взлом NTV+ с помощью распределенных вычислений

http://www.xakep.ru/post/20600/default.asp

22. Знаете ли вы, что большинство времени ресурсы компьютера используются менее чем на 5%?

http://distributed.ru/?what-is

23. Распределенные вычисления с минимальными затратами

http://msk.nestor.minsk.by/kg/2002/07/kg20708.html

24. Распределенные вычисления на FreePascal под Windows.

http://freepascal.ru/article//raznoe/20051207110629/

25. Распределенные вычисления - паразитные вычисления

http://center.fio.ru/method/resources/judina/05-03/news/parazit.htm

26. Sony рассматривает возможность продавать процессорное время PS3

http://www.gamasutra.com/php-bin/news_index.php?story=13476

27. ZLIB

http://www.zlib.org

28. NVIDIA Texture Tools 2 Alpha

http://developer.nvidia.com/object/texture_tools.html

29. hxGrid source code

http://sourceforge.net/projects/hxgrid/

30. xNormal

http://www.xnormal.net/

31. Author’s page

http:/www.deep-shadows.com/hax/

History

Version 1.09d
------------------------
- fixed critical bug: unable to complete last few tasks,
if task takes more then 20 sec;

Version 1.09c
------------------------
hxGridUser and agent are changed in this release.
- new option: 'allowDiscardCoordinatorIP' - seel hxgrid.ini for description;
- fixed bug: hxGridUser is unable to run tasks, if single task size is larger then
allowed memory usage for hxGridUser.

Version 1.09b
------------------------
- added method "IGridUser->GetConnectionStatus()";

Version 1.09a
------------------------
- added Windows 2000 support;

Version 1.09
------------------------

This is very stable release with Windows Vista support.

- fixed critical bug: memory corruption in agent;
- fixed critical bug in scktcomp.pas delphi component (thread-safety issue);
- fixed critical bug: agent is unable to load task dll in Windows Vista;
- fixed thread-safety bug in agent and griduser (AddRef()/Release() should use InterlockedXXX);
- fixed memory leak in agent (IdUDPServer not freed);
- fixed IGridUser->GetSettings() and IGridUser->SetSettings() .h headers;
- IGridUser->isComplete(var complete:boolean) now returns TRUE
when no tasks is running (was: FALSE);
- hxgriduseddll.dll updated in PICalculator\C++ example;
- fixed AutoUpgrade example: unable to upgrade
due to relevance to current directory;
- fixed memory leak in GridGMP example;
- fixed DebugWrite bug: exeption on invalid strings causes agent to lost connection;
- fixed possible thread-safety issue in agent.filecache;
- new method: IAgent->GetSessionCacheDirectory();
- new method: IAgent->GetGlobalCriticalSection();
- new IGridUser setting: failed_agent_suspend_timeout;
- updated TAgentSettings in IAgent.h;
- IAgent.h and I_Agent.pas documented in English;
- Normalmapper_hxGrid_Setup installs 3ds MAX and Maya NMF export plugins;


Version 1.08a
------------------------
- article translated to English;
- headers are documented in English;
- cancellation support;
- fixed bug in IAgent->GetData();
- fixed bug: freezing on 99.99%;
- fixed bug: lost tasks if IAgent->GetData() failed;
- fixed bug: hxgrid application is unable to work,
if workstation has not been rebuted more than 25 days;
- settings tuned for large IAgent->GetData() size;
- fixed GridGMP sample (all drawing is done in main thread);
- network and memory overload tracking: at the start of the session,
too many agents request data from user. Library tracks network and memory load;
- GridGMP example shows cancellation support and progress display;
- ATI Normalmapper installer installs NMF export plugins
to plugin directories of 3DS MAX and Maya;
- more accurate progress display in ATI Normalmapper;
- NMFExport plugins for Maya 7.0, 8.0 and 8.5 included;
Note: agent and coordinator are modified, but have the same interace versions.
Upgrade recommended (see Examples\Autoupgrade\);

Version 1.08 beta
------------------------
- initial public release;

You must Sign In to use this message board.

FAQ    Noise Tolerance Very HighHighMediumLowVery Low   Layout Expand Posts & RepliesThread ViewNo JavascriptNo JS + Preview   Per page 102550

Msgs 1 to 25 of 26 (Total in Forum: 26) (Refresh) FirstPrevNext

Callback issue TJUL 23:57 20 Jul '09   First of all great work you have done! I have written a small application together with hxgrid and it runs very fine in general. Getting a little more ambitious I wanted to use the Callback function to send task info back the the user. Same purpose as the message by Michael Stammberger. When the callback function is used the agent does not shut down when the task is done so no finalize is called and Iscomplete returns false. Callback transfer the data OK. If callback is not used the finalize and IsComplete works just fine. I use delphi and from the user program I have the settings IGridUser_Create(user); user.GetSettings(Settings); Settings.sendDublicateTasks:=false; Settings.maxSendAheadTasks:=1; user.SetSettings(Settings); user.BindGetDataCallback(callback); procedure callback(dataDesc: pchar; var stream: IGenericStream); cdecl; begin str:=strpas(datadesc); showmessage('Callback called: '+str); stream:=TGenericStream.Create(); end; In the runtask procedure, the callback is used in the following way Agent.GetData(sessionID,dataDesc,dummyStream); pointer(dummyStream):=nil; Agent.FreeCachedData(sessionID,dataDesc); Can you help me with this one? Sign In·View Thread·PermaLink Re: Callback issue hax_ 5:12 22 Jul '09   Please send me repro-case to hax@deep-shadows.com, and I will check it. You probably need to check everything on Agent side. Create grid with single agent, and attach to agent process with Delphi to debug task. Sign In·View Thread·PermaLink source code vanhutten 11:12 22 Dec '08   where can i find subj? Sign In·View Thread·PermaLink Re: source code hax_ 22:28 22 Dec '08   http://sourceforge.net/projects/hxgrid/ Sign In·View Thread·PermaLink Re: source code vanhutten 22:54 22 Dec '08   but there is only binaries, not source code, wroted on C++\C# :( Sign In·View Thread·PermaLink Re: source code hax_ 23:56 22 Dec '08   Please press "More", "CVS", "CVS Browse" buttons on sourceforge page. Full source code and examples are available in CVS repository of the project. Sign In·View Thread·PermaLink 4.00/5 (2 votes) run agent and coordinator as services Michael Stammberger 9:56 29 Oct '08   Hi Thank you very much for this great tool - I like it very much! At the moment the agents and the coordinator autostart after an user-login. What about the idea to install the agents and the coordinator as services on the machines. A login would be unnecessary, the computer has only to be switched on. I have solved this now by hand using the instsrv.exe and srvany.exe from the "Windows Resource Tools Kit". A very useful feature and improvement would be to give the user the possibility to decide at installation to install the programs as services. And/or to provide settings, where this could be set-up. Regards Michael Sign In·View Thread·PermaLink Receiving data from task while processing? Michael Stammberger 5:37 21 Oct '08   Hi is it possible to receive data from a still running task? I want to get the state or progress of the running tasks, e.g. as a percentage value. The functionality could be something similar to the GetData() feature, but in opposite direction. As an idea: A callback function on user side called periodically by the tasks running on an agent machine, sending a stream containing some data the user software could use for updating a progress bar. Best regards Michael Sign In·View Thread·PermaLink Re: Receiving data from task while processing? hax_ 2:28 27 Oct '08   First of all, hxGrid is designed to work in the following way: 1) spawn 100 tasks. 2) wait for completion. N tasks completed -> N% done. You still can use 'long task' scenario, by using "global data request callback" to pass progress information to user. Request global data with name "task XXXX, M% done" on agent, and parce tags in GetData() callback on user. Return dummy data from callback. Sign In·View Thread·PermaLink Re: Receiving data from task while processing? Michael Stammberger 1:05 31 Oct '08   Thank you very much for your answer. Concerning your suggestion to pass the progress information to the user using the global data request: I'm not sure if I understood you in the rigtht way. I tried to pass the data with GetData() using as dataDesc a string containing my progress information. The GetDataCallback then parses the string to extract the progress information. Finally it gives back a dummy stream as mentioned (containing only an int). So far so good. But for any reason the GetDataCallback is called only the first time. All following calls are ignored. Any idea? Perhaps (but unlikely) the GetData() function could only be called one time per task? Additional question: In the articel you described the function FreeCachedData(): virtual HRESULT __stdcall FreeCachedData(DWORD sessionId, const char* dataDesc); "Agent is caching global data, requested during session. This method allows to free cache and minimize memory usage, if it is known, that data will not be requested again during session." Does this mean, that for each new dataDesc (containing the progress information described above) a new global data block would be created? And do I have to free each one separately with FreeCachedData()? Yes, I know that to split the "long" task into many "short" tasks would be the best way. Unfortunately: I've to do some (up to 10) big calculations in parallel where each one needs up to approximately 3 minutes. The effort to split these into more smaller tasks is due to the effort (some parts were not written by me) not possible at the moment. Best regards Michael Sign In·View Thread·PermaLink Re: Receiving data from task while processing? hax_ 1:46 31 Oct '08   You should pass unique "dataDesc" string each time. Assumming you are using "datadesc" = "task: M, Done: N%", "datadesc" is unique for each N. Agent is caching global data, so for identical strings, GetData() callback on user are not called. It is advised to call IAgent->FreeCachedData() for free dummy stream after calling IAgent->GetData(). After this call, IAgent->GetData() will call GetData() callback on user even for identical strings. 1) IAgent->GetData("Done 0%); //callback is called on user IAgent->GetData("Done 0%); //callback is NOT called on user IAgent->GetData("Done 0%); //callback is NOT called on user IAgent->FreeCachedData("Done 0%); IAgent->GetData("Done 0%); //callback is called on user IAgent->GetData("Done 0%); //callback is NOT called on user IAgent->GetData("Done 0%); //callback is NOT called on user 2) IAgent->GetData("Done 0%); //callback is called on user IAgent->GetData("Done 1%); //callback is called on user IAgent->GetData("Done 2%); //callback is called on user Sign In·View Thread·PermaLink Where can we find xhGridInterface.h Remi Saint-Amant 11:20 3 Oct '08   Sound greats!! Where can we find xhGridInterface.h and examples? Thanks Sign In·View Thread·PermaLink Re: Where can we find xhGridInterface.h hax_ 0:40 8 Oct '08   In sourceforge repository, http://sourceforge.net/projects/hxgrid/ (more, CVS browse) Sign In·View Thread·PermaLink Good article! Damir Valiulin 12:31 27 Sep '08   Thanks for sharing this code! I can't wait to try it out. I wish you would elaborate more on the methods of the IGenericStream, but I guess I just need to look at source code. Regards, Damir Sign In·View Thread·PermaLink sounds cool! Puffel 17:27 26 Aug '08   This sounds really cool. Hope I can try it sometime soon. Sign In·View Thread·PermaLink Well done Ralph Willgoss 14:20 26 Aug '08   Great well laid out article and amazing content, keep up the good work! Sign In·View Thread·PermaLink Hadoop _Dals_ 12:19 25 Aug '08   Why not hadoop? Sign In·View Thread·PermaLink Re: Hadoop hax_ 13:14 25 Aug '08   As I said, we can't afford large dedicated cluster. At the same time, we have 50+ office PCs running Windows XP, which processing power is used by less then 10%. I have not found any grid platform which can use IDLE TIME OF OFFICE PC's. hxGrid is designed to work as silent as possible. From the year of experience, no single user complained slowdowns, while grid was used intensively. The most closest solution is Xoreax XGE engine, but it just did not exist 1,5 years ego. p.s. Besides Alchemy, there is also interesting MC# language: http://u.pereslavl.ru/~vadim/MCSharp/index.php Sign In·View Thread·PermaLink awesome! alejandro29A 11:01 25 Aug '08   thanks! Dios existe pero duerme... Sus pesadillas son nuestra existencia. (Ernesto Sabato) Sign In·View Thread·PermaLink Re: awesome! Fernando Vizer 22:44 25 Aug '08   Sabato, me desperte, y te estoy esperando... Dios Fernando Vizer Enterprise Architect Sign In·View Thread·PermaLink Re: awesome! alejandro29A 3:30 26 Aug '08   ajjajaj es mi firma (ese libro es genial) ajjajaj it's mi signature (that book is great) Alejandro Varela Dios existe pero duerme... Sus pesadillas son nuestra existencia. (Ernesto Sabato) Sign In·View Thread·PermaLink Re: awesome! Fernando Vizer 4:54 26 Aug '08   Hola, Encantado, soy argentino, pero vivo muylejos de la patria. Como esta avanzada la cosa del cloud ( o grid) donde trabajas ? Yo estoy en haciendo un research, Fernando HP Software Fernando Vizer Enterprise Architect Sign In·View Thread·PermaLink Re: awesome! alejandro29A 5:18 26 Aug '08   buenisimo! te cuento, programo en C, vivo en rosario (santa fe), trabajo para una empresa de colectivos (hago un programita que entre otras cosas sigue a los colectivos con un gps). me interesa mucho el tema del diseño de lenguajes. queda abierto el espacio para la charla. te dejo mi mail! hav29a @ gmail.com Dios existe pero duerme... Sus pesadillas son nuestra existencia. (Ernesto Sabato) Sign In·View Thread·PermaLink Re: awesome! Fernando Vizer 6:44 26 Aug '08   Hola Alejandro, A mi me gustaria hacer programas mas tangibles, como el que describis, y menos teoretico, peroo la realidad obliga... Hace mucho que no veo un bondi, ni en figurita. Rosario, wow, la ultima ves que estuve alli fue por el '79, tenia unos diesisiete. Tenia unos familiares que vivian cerca de la avenida circunvalacion, a unas cinco cuadras al sur de una estacion de carga de tren El barrio no me lo acuerdo, puede se que vos sepas. Disenio de lenguajes ? Solo en la UBA ( jyyac se llamaba ) En mi vida profesional programe en Cobol, RPG, (seis anios), depues en C, hasta 1998, despues en C++ , hasta 2002, y por ultimo en JAVA, hasta hoy. Vivo en Israel desde 1990, con algunas lagunas en USA y Europa, por medio de diversas companias. Desde hace unos cinco, hago mas disenio que programar, aunque los POC los hago yo mismo ( hands on). Como esta el mercado , tanto laboral como tecnico en la Argentina ? Fernando Fernando Vizer Enterprise Architect Sign In·View Thread·PermaLink Re: awesome! alejandro29A 8:11 26 Aug '08   acá está bastante bien el mercado laboral. piden mucha gente. mejoró mucho desde el 2003 (por la crisis del 2001) pero siguen sin pagar como en otros países, yo por mi parte estoy bien, pero no es muy común que se les pague bien a los programadores por estas latitudes. yo tengo 28 años por cierto, veo que sos mas grande que yo retomando el tema, esta todo mucho mejor por acá, pero siempre pasa algo y viene alguna crisis.. Dios existe pero duerme... Sus pesadillas son nuestra existencia. (Ernesto Sabato) Sign In·View Thread·PermaLink

Last Visit: 5:30 21 Nov '09     Last Update: 5:30 21 Nov '09 12 Next »