Organic Programming Environment (OPEN)

Introduction

GUI applications are event driven, in that they respond to user and hardware events. These events perform one or more of the following processes:

1. Data collection;
2. Data transformation;
3. Data dispersion.

Collecting data includes such activities as:

1. Reading information out of a database;
2. Storing a GUI object?s value in a variable;
3. Reading a hardware register.

Data transformation consists of any process that manipulates the data, combining, splitting, formatting, etc. Data dispersion is the opposite of data collection, during which data is written to a database, a GUI object is updated, or a hardware register is written.

In this paradigm, data is usually very localized, contained within a single function or managed by a single object. Interface methods of various complexity exist to share data between objects. Ultimately, a large application must manage a diverse amount of data and perform complicated tasks on this data. This leads to complexities in managing various aspects of data:

1. Lifetime;
2. Version / Format;
3. Read/Write access.

These tasks are left to the programmer, and are in my experience mostly ignored.

In the Organic Programming Environment (OPEN), a data-centric approach is taken. A Data Pool (D-Pool) manages all data. Processing functions register themselves with the D-Pool Manager (DPM), indicating the data on which they operate, and the data that they produce. When data is placed into the D-Pool, the DPM automatically initiates all interested parties as threads. When all threads have completed, the data is automatically removed from the D-Pool. The DPM can also be instructed regarding data lifetime, version/format information, and read/write access.

The Name

I came up with this name because I wanted to describe a process similar to how proteins are transported out of the nucleus of a cell and manipulated into more interesting molecules, that are then either used by the cell itself or transported out of the cell, to be used by some other cell or organ. This process seems to me to be inherently data-centric, with the DNA/RNA being the data, and various processes (enzymes, etc) attaching themselves to the data as it appears in the cellular fluid. Please note that this architecture is not based on either genetic programming (http://www.genetic-programming.org/) nor the Organic Programming Language Gaea (http://www.carc.aist.go.jp/gaea/).

Benefits

There are several benefits to this architecture:

1. Processes are completely isolated from each other, improving portability and reuse;
2. Each process clearly documents the data types in which it is interested;
3. The operation of the program is completely traceable without the programmer having to remember to add trace messages. The DPM can handle all tracing of data and process invocation;
4. Since each process is invoked as a thread, the program runs as optimally as possible;
5. The programmer is completely alleviated from having to think about whether a thread would improve performance and doesn't need to consider most issues of data synchronization.

Drawbacks

There are also several drawbacks to this architecture (I'm sure I haven't thought of them all!)

1. Programs become non-linear in their operation, which is difficult to debug;
2. Sequences of events may occur in a different order from one run to another, again making the program difficult to debug;
3. Because the datum must be managed by a key, or name, it is possible to have name collisions? In any case, even trivial information must be named, which can become a cumbersome task;
4. Globalizes a lot of data that doesn't necessarily need to be treated as global;
5. Data dependencies, especially when updating / inserting database records, are difficult to support when the database architecture includes foreign key associations (which any robust database design should implement) [sorry MySQL!])
6. There is no support for this kind of modeling in existing development frameworks such as MFC;
7. Existing frameworks, such as MFC, must be significantly enhanced to incorporate this architecture;
8. Most importantly, it is a completely foreign way of thinking about programming, and therefore will probably not be widely considered as practical.

Design

There are four design elements of OPEN:

1. The process pool (P-Pool);
2. The data pool (D-Pool);
3. The data pool manager (DPM);
4. The data collection container.

This architecture will quickly create hundreds, if not thousands, of small processes that receive some data, manipulate it, and place the result back into the D-Pool. Registering all these processes manually becomes a very cumbersome task, so it behooves us to make this as simple for the programmer as possible.

There process must implement three things:

1. Registering the process itself;
2. Registering the data on which the process operates.
3. Providing the actual implementation of the process.

Each process can operate on more than one datum. If, for example, a process uses two data, the DPM must check every permutation of datum currently in the pool to determine if a process should be triggered. This will lead to disaster as the number of datum increases in the D-Pool. To avoid this, a specialized container is implemented that collects datum. When the collection is complete, the process is triggered. This collection is implemented as an STL multi-map. A datum can be associated with one or more collections-processes. The collection acquires the same name as the collection's process, thus associating the collection with its process. When a datum is placed into the D-Pool, the DPM iterates through the multi-map of datum, adding the datum to each associated collection. Every process whose collection has a completed datum list is then triggered.

This design has undesirable side affects:

1. A particular datum in a collection can change before the collection is complete;
2. Clearing the datum before the process exits can result in throwing away datum already being collected for a second iteration of this current process.

For this prototype, these side affects are ignored.

Finally, the D-Pool does not know what the data type is, for each datum in the pool. Using templates is not feasible because templates require the type to be known at compile time, imposing very annoying requirements on the programmer. Instead, the D-Pool maintains a collection of generic data containers. The data container is smart enough to convert from and to various built-in types and includes a virtual base class for custom derivations. Since the data container is not really the topic of this paper, its design and implementation is ignored. Feel free to browse through the source.

Implementation

To implement this design, all processes must be declared as classes and derived from the OPEN_Process class.

class OPEN_Process
{
public:

    virtual ~OPEN_Process() {}

    void RegisterDNames(void);
    bool SetData(const CString& dataName, const DataContainer& dc)
    {
        dataNameList[dataName]=dc;
        // this is quite the kludge to see if all datum for 

        // the process has been set

        return nameList.size() == dataNameList.size();
    }

    virtual void Run(void)=0;

protected:

    OPEN_Process(const CString& s1, const CString& s2) : 
         pName(s1), dName(s2) {}
    OPEN_Process(void) {};
    OPEN_Process(const OPEN_Process& p) :
    pName(p.pName), dName(p.dName), nameList(p.nameList),
        dataNameList(p.dataNameList) {}

protected:

    CString pName;
    CString dName;
    std::vector<CString> nameList;
    std::map<CString, DataContainer> dataNameList;
};

This class is implemented as a virtual base class. Other than the constructor (which must be invoked from a derived class), the only method of real interest to the user is Run(void), which must be implemented in the derived class. Note also that this class encapsulates the process name, the original datum name list, the datum name list separated out into a vector, and an STL map associating the datum name with the container that maintains the actual value. This information can be used to generate debugging information or a data diagram (for example, in Visio) of the data flow. Describing output data is currently not a requirement but could be easily added.

Each process is managed by a process pool. The OPEN_ProcessPool essentially encapsulates an STL map that associates the process name with a pointer to the process. Other objects in OPEN also use this class to interface to a specific process object.

class OPEN_ProcessPool
{
public:

    OPEN_ProcessPool(void) {}
    virtual ~OPEN_ProcessPool() {}

    void Register(const CString& processName, 
        class OPEN_Process* proc)
    {
        processList[processName]=proc;
    }

    bool SetData(const CString& processName, const CString& dataName, 
        const DataContainer& dc)
    {
        ASSERT(processList.find(processName) != processList.end());
        bool trigger=processList[processName]->SetData(dataName, dc);
        return trigger;
    }

    void Trigger(const CString& processName)
    {
        ASSERT(processList.find(processName) != processList.end());
        AfxBeginThread(OPEN_ProcessPool::StartProcess, 
            processList[processName]);
    }

protected:

    static UINT StartProcess(void*);

public:

    static OPEN_ProcessPool pool;

protected:

    std::map<CString, class OPEN_Process*> processList;
};

There is no particular reason from the programmer to interface to this class directly.

The OPEN_DataCollection implements the one to many association between the datum name and the processes that are interested in that datum. This is implemented as an STL multimap, and this class is essentially a wrapper for the multimap, providing registration and iteration methods.

class OPEN_DataCollection
{
public:

    OPEN_DataCollection(void) {}
    virtual ~OPEN_DataCollection() {}

    void Register(const CString& datumName, const CString& collName)
    {
        collectionList.insert(std::pair<const CString, 
            CString>(datumName, collName));
    }

    bool FindFirst(const CString& dataName, CString& collName)
    {
        iter=collectionList.find(dataName);
        ASSERT(iter != collectionList.end());
        collName=(*iter).second;
        return iter != collectionList.end();
    }

    bool FindNext(const CString& dataName, CString& collName)
    {
        ++iter;
        if (iter==collectionList.end())
        {
            return false;
        }
        collName=(*iter).second;
        return (*iter).first == dataName;
    }

public:

    static OPEN_DataCollection coll;

protected:

    std::multimap<const CString, CString> collectionList;
    std::multimap<const CString, CString>::iterator iter;
};

The OPEN_DataPool implements a wrapper around yet another STL map. This map associates the datum name with the actual value. It is this object with which the application interfaces to place data into the D-Pool. This class also implements a semaphore that unblocks the DPM, which then parses through the data in the D-Pool, removing it and placing it into the corresponding process container. This class also implements a CRITICAL_SECTION to ensure that the DPM (running as a thread) can read and write the collectionList without colliding with other threads that are potentially writing and erasing this collection also.

class OPEN_DataPool
{
public:

    OPEN_DataPool(void)
    {
        InitializeCriticalSection(&cs);
        dpSem=CreateSemaphore(NULL, 0, 0x7FFF, "OPEN_DP_SEM");
        ASSERT(dpSem);
    }

    virtual ~OPEN_DataPool()
    {
        DeleteCriticalSection(&cs);
        CloseHandle(dpSem);
    }

    void Add(const CString& dataName, const DataContainer& dc)
    {
        EnterCriticalSection(&cs);
        dataPoolList[dataName]=dc;
        LeaveCriticalSection(&cs);
        ReleaseSemaphore(dpSem, 1, NULL);
    }

    void RemoveDatum(CString& s, DataContainer& d)
    {
        EnterCriticalSection(&cs);
        std::map<CString, DataContainer>::iterator iter
            = dataPoolList.begin();
        ASSERT(iter != dataPoolList.end());
        s=(*iter).first;
        d=(*iter).second;
        dataPoolList.erase(iter);
        LeaveCriticalSection(&cs);
    }

public:

    static OPEN_DataPool pool;

protected:

    std::map<CString, DataContainer> dataPoolList;
    CRITICAL_SECTION cs;
    HANDLE dpSem;
};

The OPEN_Mgr implements the DPM. This class is very simple:

class OPEN_Mgr
{
public:

    OPEN_Mgr(void) {}
    virtual ~OPEN_Mgr() {}

    void Run(void);

public:

    static OPEN_Mgr mgr;

protected:

    HANDLE dpSem;
};

What is of more interest is the implementation of the Run method:

void OPEN_Mgr::Run(void)
{
    dpSem=OpenSemaphore(SYNCHRONIZE, FALSE, "OPEN_DP_SEM");
    ASSERT(dpSem);
    while (1)
    {
        DWORD ret=WaitForSingleObject(dpSem, INFINITE);
        if (ret==WAIT_OBJECT_0)
        {
            CString dataName;
            CString processName;
            DataContainer data;
            OPEN_DataPool::pool.RemoveDatum(dataName, data);
            bool ret=OPEN_DataCollection::coll.FindFirst(dataName, 
                processName);
            while (ret)
            {
                bool trigger=OPEN_ProcessPool::pool.SetData(processName, 
                    dataName, data);
                if (trigger)
                {
                    OPEN_ProcessPool::pool.Trigger(processName);
                }
                ret=OPEN_DataCollection::coll.FindNext(dataName, 
                    processName);
            }
        }
        else
        {
            break;
        }
    }
}

Implemented as a thread, this function waits for datum to be placed into the data pool, upon which the thread is released. It iterates through the data pool, removing each datum and value from the pool and storing it in the each process that is interested in the datum. The DPM then "triggers" the process as a thread when all the datum that the process requires has been instantiated.

To support a somewhat more readable implementation of processes, several macros are defined:

#define DECLARE_OPEN(x, y) \
class x : public OPEN_Process \
{ \
    public: \
    x(void) : OPEN_Process(#x, y) {} \
    virtual void Register(void) \
    { \
        RegisterDNames(); \
        OPEN_ProcessPool::pool.Register(#x, this); \
    } \
    virtual void Run(void); \
    static x _##x; \
}; \
x x::_##x;

#define IMPLEMENT_OPEN(x) \
void x::Run(void) {

#define FINISH_OPEN \
dataNameList.erase(dataNameList.begin(), dataNameList.end()); }

#define REGISTER_OPEN(x) \
x::_##x.Register()

Thus, the implementation of a process would look something like this:

DECLARE_OPEN(AddCost, "itemCost");
IMPLEMENT_OPEN(AddCost)
{
    double cost;
    dataNameList["itemCost"].Get(cost);
    double total=atof(dlg->total)+cost;
    OPEN_DataPool::pool.Add("totalCost", 
        DataContainer().Set(AutoString(total)));
}
FINISH_OPEN

And somewhere in the initialization section of the application, the process must be instantiated:

REGISTER_OPEN(AddToList);

The issue of process instantiation is an annoying one. You will notice that almost all of the OPEN classes automatically instantiate a singleton, implemented as a public static member of each class. This is also done with the processes. However, the actual registration of the datum names cannot be done at program startup because other necessary initialization (for STL, for example) has not occurred. To my knowledge, there is no way of pre-determining the initialization order of global or static data.

The Demonstration Program

The demonstration program is a simple and rather poor example of how this paradigm works. The "Add Part", "Remove Part", and "Clear Part" events each place data into the D-Pool which is picked up by the DPM. The most interesting thing about this example is that the datum "itemCost" triggers two events: one to update the list box and the other to update the running total. In actuality, this example is a poor one because the process handlers are so tightly coupled to the dialog object, which is definitely not a desirable thing to have happen in real life. Also note that I over-rode the AssertValid method so that I could update the dialog from a worker thread while still running in debug mode. Good 'ol MFC.

A More Interesting Thought Experiment

For this model to really be effective, the developer needs to completely rethink how applications are designed and implemented.

For example, a function might do something like this:

• DB query Q1
• DB query Q2
• Operation A
• DB query Q3
• DB query Q4
• Operation B
• DB update QQ
• DB update RR

In the spirit of this architecture, the function should be recoded into several processes:

Process 1:

DB query Q1

Process 2:

DB query Q2

Process 3: (depends on Q1 and Q2)

Operation A

• Output result R1 into Data Pool

Process 4: (depends on R1)

DB query Q3

Process 5: (depends on R1)

DB query Q4

Process 6: (depends on Q3 and Q4)

Operation B

• Output result R2 into the Data Pool
• Output result R3 into the Data Pool

Process 7: (depends on R2)

DB update QQ

Process 8: (depends on R3)

DB update RR

As you can see from the above architecture, the program now automatically performs the database queries and updates simultaneously in separate threads. This can dramatically improve program performance and it is achieved by simply using a different paradigm for data management.

The Challenge

I consider this paradigm a significant enhancement to the existing process-centric programming styles. It results in:

1. smaller and simpler functions;
2. greatly increases parallel processing;
3. automatically provides detailed function and data tracing;
4. and documents all function inputs and outputs.

There are complexities in this model that are not fully understood. My challenge to the reader is to identify these complexities and design solutions for them, ultimately making this paradigm robust and easy to use. For example, in this prototype implementation, the DPM, D-Pool, and other objects are implemented as global singletons. It seems instead more reasonable that an application would have several data pools at different scales. This would extend the entire concept of organic programming? For example, (pardon the analogy), program organs.

Credits

Please credit the author, Marc Clifton, for the core of OPEN in any application that you build using it (I'm probably dreaming, right?). The author (me) also requests that he is provided with the source code and list of all enhancements that you make to the architecture of OPEN, so that they may be included in future versions for the benefit of all.

Conclusion

The Organic Programming Environment is not a replacement for process-centric modeling. However, the OPEN is a significant enhancement in the programmer's toolset because there are many cases where a data-centric model is superior to a process-centric one.

You must Sign In to use this message board.

FAQ    Noise Tolerance Very HighHighMediumLowVery Low   Layout Expand Posts & RepliesThread ViewNo JavascriptNo JS + Preview   Per page 102550

Msgs 1 to 25 of 35 (Total in Forum: 35) (Refresh) FirstPrevNext

Datum NoUserNameFound 2:16 9 Jul '04   Hello, I like the concept presented here,though I still need to go thru it in more detail ( Also I am quite keen reading thru the AAL articles which I believe will extend the concept presented here ) I would like to get the clarifications on 2 things a) It was mentioned in one of the messages that Datum can also hold a "complex type" say another object. However won't that go against what you are trying here, as "Object" is nothing but collection of simple types with constraints ( like ultimately say myobject will be composed of say int age int, string name exposed thru their properties ). If we disperse these properties into individual datums won't that be more useful in this model? b) I also feel we should associate Datum with "rules" to ensure it always has a valid state. I was trying to come up with somthing similar where I maintain an associated global pool of data and a pool of operations, application acts as a builder to various operations associating it with a particular view of the "data" sac666 Sign In·View Thread·PermaLink scalability, loops, and OS support Keith Farmer 10:23 3 Nov '03   On WinFS-based systems, coding this sort of environment seems like it should be almost trivial. The use of Intelligent Agents to monitor a data store and trigger recalculations could be the canonical way to implement this sort of environment, perhaps? Two questions: WinFS or not, how lightweight can this system be made? Scaling, perhaps, to millions of data points? I've been planning a similar scheme for a distributed astronomy project in C#. Is there an efficient scheme to prevent calculation loops? .... Always having to bookmark you! Sign In·View Thread·PermaLink Re: scalability, loops, and OS support Marc Clifton 14:51 3 Nov '03   Keith Farmer wrote: how lightweight can this system be made? I'm not sure. I think the problem with a large set of datapoints comes down to two things--either use a tremendous amount of processing power to recheck the data set when something is added, removed, or changed, which must be done for each agent, or, use a large amount of memory in which each agent is notified of data changes and keeps its own collection. I've opted for the latter method, because it becomes trivial for the agent to determine whether the data set is complete--it's basically just a counter tracking how many things the agent is interested in exist in the data collection. Memory is a lot cheaper than processing power, so tracking several million data points shouldn't be too bad if you use an efficient hash algorithm for referencing the data, I guess. On the other hand, your per agent list management starts to get processor intensive--for example, if there's a million data points and one of them changes, and your agents are interested in, say, a large subset of those points, like hundreds of thousands, you need a fast way to test the agent's data list to see if it needs to do something when a change is made. Again, a good hash scheme would help, but there's probably smarter things you can do if you know the characteristics of your data. Does that make sense? What sort of distributed astronomy project do you have in mind? Thanks for the comments, BTW! Marc Latest AAL Article My blog Join my forum! Sign In·View Thread·PermaLink Re: scalability, loops, and OS support Keith Farmer 16:00 5 Nov '03   Basically, to support amateur observations -- images, annotations, measurements (scalar, vector, spectral, waveform), and derivatives of these (for example, a curve fitting of a variable-magnitude star, or statistics regarding a newly-discovered supernova's brightness curve over time). If you're familiar with AAVSO or ALPO, I think they'd be the most likely targets of this sort of system. I'm interested in tying this sort of system with a grid architecture to allow the information to be more or less federated. Based on a small-scale system that I got to work on at my previous job, I was leaning toward data keeping track of its most recent result, which it would only recalculate when notified that its source was changed. Not yet specified is whether the source is defined statically or dynamically (difference between { Io, Europa, Ganymede, Callisto } and "the 4 largest moons of Jupiter"). Statically would be easiest to chain update events, and was what was used on the system I dealt with before. However, while this may work for a handfull of interacting objects, I don't know how well it scales for a more intensive research project. Sign In·View Thread·PermaLink Sources still available? abracadaboink 17:39 3 Jun '03   Hello Marc, I've emailed you a while back for the sources, are they still available? If so, please email to abracadaboink@yahoo.com (if you have C++, that's great. If it's C#, I'll translate) Thanks! AB Sign In·View Thread·PermaLink 2.00/5 (1 vote) N-Tier TW 4:48 1 Jun '03   The process pool (P-Pool) = Application Server cluster The data pool (D-Pool) = Database Server cluster (N-Partition possible) The data pool manager (DPM) = Resource Manager The data collection container = XML is highly recommended The above simple list (hardly hit 60% of a true large system architecture) is more then enough to implement this so call "Organic Programming Environment". The key point is decompose a Domains -> Domain -> Systems -> System -> Modules -> Module -> Services -> Service -> Objects -> Object. Where any of the "noun" used here can be replace by your project, example if you are doing VxWork, RTOS32, these "noun" could be very different. But design strategy is always first decompose, and put smaller parts either in layer form, or distribute node form, and defined strategy for supporting services (cache, backup, lock, synchronize etc) at each layer/node/cluster/group/hierarchy etc. Also, in a large system, client access strategy often vary from module to module. A badly design backup plan for data tier could easily bring any data-driven system to knee. A badly credential management plan could easily halt any application server...think again, you sure to decompose a system into "pool"? I would rather recommend design a system via "scale-out" spirit (n-tier). "Organic Programming Environment" in my point-of-view, attempt to re-invent an already solved problem. Any methodology attempt to solve today's complex I.T problem, must align itself with I.T infrastructure design methodology. This is not something like plug-in the network cable, instantiate a remote object from pool, submit a query and expect a returned collection of data could meet. I also couldn't understand what does the author means by "data initiate process", interprocess communication and data collection schema (why not MDAC? or Java data tier) is simply not enough to justify "data driven", where is business logic comes? data presentation? partition? trigger? lock? sync?.... Nevertheless, good try. Try harder! *What is a true "large system"? A strategy (implement by information technology) that satisfy the global (n-culture, n-policy , n-....) needs, example human resource I.T system for ExxonMobile international etc. Sign In·View Thread·PermaLink Re: N-Tier Marc Clifton 6:09 1 Jun '03   TW wrote: "Organic Programming Environment" in my point-of-view, attempt to re-invent an already solved problem. In that case, why don't I see data driven architectures (as opposed to procedural ones) more commonly supported in frameworks? And if you read my AAL articles, why don't I see clustered data driven event mechanisms processed through an established interface mechanism. TW wrote: Any methodology attempt to solve today's complex I.T problem, must align itself with I.T infrastructure design methodology [sarcasm]Right.[/sarcasm] First off, IT's complexity varies from the trivial to the extremely complex. This architecture fits somewhere in the middle, I'd say. Second, aligning with existing infrastructure design methodology isn't very innovative, now is it? If we aligned ourselves with existing technologies, I'd have a lama in my backyard to carry my groceries. The point of this article was to get people thinking in a different way. TW wrote: I also couldn't understand what does the author means by "data initiate process", Oh. Well, I guess I'd say that if you were to stop thinking in rigid terms of existing technologies and open your mind to other ways of doing things, you might understand this concept. TW wrote: Nevertheless, good try. Try harder! Thanks for the feedback! I like your decomposition point. It reminded me of something I'd forgotten about. Marc Help! I'm an AI running around in someone's f*cked up universe simulator. Sensitivity and ethnic diversity means celebrating difference, not hiding from it. - Christian Graus Every line of code is a liability - Taka Muraoka Microsoft deliberately adds arbitrary layers of complexity to make it difficult to deliver Windows features on non-Windows platforms--Microsoft's "Halloween files" Sign In·View Thread·PermaLink Re: N-Tier TW 8:07 1 Jun '03   My understanding of "data driven" at the simplest form is: "The meaning of a piece of information decide the next step.". Hence, if you are looking for process pool architecture + data driven solution, search for UNIX based solution. But I want to stress here again, I can see you try very hard in define your OPEN, unfortunately I really don't think this is a good (or even a correct one...) approach in developing large system. I have explained why in my previous message. - One of your reader said OOP has reached its limit.. -> -> - "Declarative programming" means configurable software component. You put code together by "switching attribute" here and there. Hence, you "declare" your intends instead of writing code from scratch. Example you can integrate DCOM security by writing ::CoSecurityXYZ() manually in your COM server, but you can also inherit the default behavior and further customize to your need, by switching DCOM security attribute via DCOMCNFG utility, hence you "declare" what you want and the system carry out for you. Many UNIX process work with this methodology, too. Good day! Sign In·View Thread·PermaLink Look at Linda Spaces Michael Feldman 7:49 2 Jan '03   I agree with your premises for need of change for software development, but what you describing looks like Linda Spaces. http://iamwww.unibe.ch/~scg/Research/ComponentModels/linda.html Sign In·View Thread·PermaLink Re: Look at Linda Spaces Marc Clifton 8:06 2 Jan '03   Very interesting. Thanks for the information! I'll definitely look at it, as it applies also the the Application Automation Layer I'm building. Cool! Marc Help! I'm an AI running around in someone's f*cked up universe simulator. Sensitivity and ethnic diversity means celebrating difference, not hiding from it. - Christian Graus Every line of code is a liability - Taka Muraoka Sign In·View Thread·PermaLink An interesting parallel Rob Graham 18:10 23 Dec '02   You might be interested in looking at the LabView programming environment for something that closely parallels your pardigm. LabView[^] works on a "DataFlow" execution model "program nodes" (primitives operators, subroutines and subprograms) only execute when all of the "data inputs" are satisfied by the presence of new data (values unimportant)... The programming environment is entirely graphical, with data connections shown as "wires" connecting iconic Function objects. It, however , exhibits many of the same probles that you anticipate: program execution sequence is not always predictable, forces lots of data globals that might not otherwise be needed ...etc. Some ideas are so stupid that only an intellectual could have thought of them - George Orwell Sign In·View Thread·PermaLink Re: An interesting parallel Marc Clifton 1:44 24 Dec '02   Very interesting. Thanks for the info! I remember LabView from years ago when I was working with some hardware engineers. Funny how things come back around. Marc Help! I'm an AI running around in someone's f*cked up universe simulator. Sensitivity and ethnic diversity means celebrating difference, not hiding from it. - Christian Graus Every line of code is a liability - Taka Muraoka Sign In·View Thread·PermaLink Request for Source Codes sigtrans 16:52 23 Dec '02   Can you send me the source codes for this? My Email: sigtrans@nownuri.net. Thank a lot! Bye Max Sign In·View Thread·PermaLink Why reinvent the wheel ? _Shark_ 5:00 2 Jun '02   What you are trying to implement here is the declarative programming model (it's very easy to see that your processes are rules that apply on data sets). So why not use something that was really designed for this (using proper pattern-matching and inference algorithms) ? CLIPS language has been around for almost two decades, and does the job just perfectly (it can also be interfaced with other high level languages). http://www.ghg.net/clips/CLIPS.html This might give you a new perspective over you current projects, too. Steve. PS. Don't get me wrong. This stuff is interesting, but to get it into practice, there is alot to do ... so why not use a system that's been proven to work ? Sign In·View Thread·PermaLink Re: Why reinvent the wheel ? Marc Clifton 11:24 21 Jun '02   This is a long time in reply, but here goes. I did some investigating on what you mention, and I found interesting. However, CLIPS touts itself as an expert system. Looking up declarative programming, I found several definitions that aren't really in sync. Microsoft even claims that COM is a declarative programming model. In any case, I disagree that I'm re-inventing the wheel. I'm not creating an expert system, which is based on the _value_ of the data. Nor am I re-inventing COM, or any other event callback system. These systems are usually based on user initiated events, not data creation events. What I am interested in exploring is an event driven component based on the _existence_ of data and their combinations, regardless of _value_. Although, I suppose what I've done here is a type of expert system and a kind of event callback COM approach. Maybe the difference is in the details. But I haven't come across a good, concise definition of the declarative programming model. Any good one or two sentence definitions lying around? Marc Sign In·View Thread·PermaLink Request for source Steve Roach 23:15 28 May '02   Marc, I'm interested to see any code that supports these ideas. Please can you send it. cheers Steve Sign In·View Thread·PermaLink Very interesting... Matt Gullett 3:07 28 May '02   I enjoyed reading your article and it is refreshing to see someone thinking "out of the box". I have not downloaded the code yet, but I have several questions (if you don't mind.) How does a process describe its data? Is is simple values or complext types? How is data persisted in the data pool? Is it in memory or in a database, XML? Can your organic model be contained within a parent program? (ie. can I have multiple shells as in one gui and 1+ console apps.) OR, is this more of a service running stand-alone? I can see some advantages to this way of thinking especially in areas of back-end data processing. Segmenting the work into small chunks can be very effective if done correctly. I have done something similar to this on a previous project. Sign In·View Thread·PermaLink 2.00/5 (1 vote) Re: Very interesting... Marc Clifton 4:12 28 May '02   Thanks for your feedback! How does a process describe its data? In the prototype, this is done as simple as possible: for example, to register a process that is interested in only one data element: DECLARE_OPEN(AddCost, "itemCost"); A process that is interested in several data elements uses the pipe '|' character to separate the names: DECLARE_OPEN(AddToList, "partNum|itemCost"); Some comments about type information: Type information is completely ignored. Each Datum is stored in a generic DataContainer. For primitive types, on the first "setter", the DataContainer records the data type as the "master" type. The "getter" functions try to convert from the master type to the requested type as best they can. Subsequent "setter" functions of a different type also will convert to the master type. I use this class extensively in my development because I frequently need to convert between different data representations, especially between database, internal, and GUI representations. Of course, there are all sorts of arguments about strong/weak typing that I'm ignoring! The DataContainer can also manage structures and classes, derived from a virtual base class, but then the "getter" needs to downcast to the required type. Thus, complex types can also be handled. How is data persisted in the data pool? Being a prototype (I keep emphasizing this ), the data pool is exclusively in memory. Definitely many possibilities here. It would be very easy to import/export a snapshot of the data pool to a database for primitive types. However, keep in mind that data usually doesn't linger very long in the data pool because the DPM reads it out and passes it to the interested processes. Data lifetime is definitely an interesting topic of discussion with this model. Can the model be contained within a parent program? Currently not. I once developed a similar system that shared data across applications. A lot of these issues are handled by existing frameworks, and instead of inventing the wheel again, I would like to interface to one of those. Any suggestions? There seems to be a lot of players out there--XML, COM/DCOM, CORBA, shared memory, etc. Hope this answers your questions! Marc Sign In·View Thread·PermaLink Re: Very interesting... Matt Gullett 6:51 28 May '02   Thanks, you answered all my questions pretty well. I'm very interested in your processing model because I can visualize a way to integrate it with a system that I have been developing the past 3 years. Marc Clifton wrote: Type information is completely ignored. This is one of the major advantages of the system I have been developing because it allows for run-time configurable type enforcement (w/automatic type conversion where needed.) Also, my system allows for complex multi-layer/tier data structures (again run-time configurable) which eliminates the need for each process to be transparent to the others. Marc Clifton wrote: the data pool is exclusively in memory Again, having runtime customizable data types means that I can create my own custom persistince systems. We can persist to memory, XML, relational database, flat file, etc. I digress. I guess the point I was trying to make is that combining your processing model with my data model I could see a lot of possibilities. Keep up the great work. I look forward to reviewing your code in more detail. Sign In·View Thread·PermaLink Re: Very interesting... Marc Clifton 11:03 29 May '02   Just out of curiosity, when exporting to a database, are you able to store the data in a way that takes advantage of the relational capability of the data, or are you really just saving it in a flat file format? Marc Sign In·View Thread·PermaLink Re: Very interesting... Matt Gullett 11:53 29 May '02   Marc Clifton wrote: when exporting to a database Actually we typically use a relational database as the physical storage medium. The system allows for physical data models which rely on PKs, RI, normalization, constraints, indexes, etc. We can use flat files, XML, IOstreams, MFC CArchive, etc for physical storage but in most situations a relational dataabse is ideal for storing the raw data. We can import/export (not really a good term for our system, we persist data) to/from any of the various physical storage mediums we support. That's probably more of an explanation than you wanted, so sorry. Sign In·View Thread·PermaLink Concurrency problems ? Daniel Turini 1:32 28 May '02   Nice article, it made me thinking about how we design applications. Ok, I programmed in the last 20 years and saw structured programming hitting its limits and the birth of OOP. These days, I clearly saw that structured programming was "dying", but could never think of OOP until I saw it. Nowadays, I see some signals of what I saw ~15 years ago: OOP seems to be hitting its limits. The projects now seems to be so big that is hard to manage all those classes and instances "floating around". [Do not flame me, please, it is hard, not impossible.] IMHO what is needed is another paradigm (I hate this word) shift. And I agree with you that a data-centric approach seems a nice way to solve several things today. Any new paradigm will have to deal with things like scalability and code reuse. OPEN seems to lack strong locking semantics and still leaves the problem open to the programmer model it. I don't know if I am being clear right now, I know I only have the problem, not the solution, but I would like a way of thinking that simplifies parallel problems. For me, today, the only hard to debug problems are the multithreaded ones. These beasts took me away several sleep nights in the last 2 or 3 years. Multithreaded code can easily become spagetthi code. Worst: OOP spagetthi code. It's hard creating easy-to-read and manageable multithreaded code. Maybe the solution to this already exists, but I can't see it, just like when OOP appeared. Crivo Automated Credit Assessment Sign In·View Thread·PermaLink Re: Concurrency problems ? Marc Clifton 2:28 28 May '02   I whole-heartedly agree, and yes, there are lots of classes/instances floating around in large programs. In applications I've written, I've been very concerned with the interdependencies of objects. (Nowadays, I tend to use classes to implement very foundational concepts that most likely won't change over the project development, and I use a script based architecture for everything else.) I've observed that multithreading, while powerful, is poorly used. Part of the problem, I think, is that other than a lot of primitives for controlling data access, there's no higher level architecture that makes data/thread management more "automated". Microsoft has put tons of work into GUI, component, and web-based architectures, but nothing other than OS primitives in thread/data management issues. I think this whole issue needs addressing, and while, as you say, OPEN doesn't provide strong locking semantics, this issue is definitely centered on my "scope", as it were. Thanks for your feedback! Marc Sign In·View Thread·PermaLink This is interesting Infinitus 23:47 27 May '02   "there are many cases where a data-centric model is superior to a process-centric one." Right you are! I have used a similar model for some time now, but have never put down a description of it. Well, I don't have to bother now Keep up the good work - long live OPEN Sign In·View Thread·PermaLink Problem With Source Marc Clifton 12:06 27 May '02   Hello everyone, This was my first posting and I goofed on the source code. If anyone is interested, email me directly and I'll send you the zip file. Marc Clifton Sign In·View Thread·PermaLink

Last Visit: 19:48 9 Jul '09     Last Update: 19:48 9 Jul '09 12 Next »