Integration: Kinematics + Digital Image Processing + 3D Graphics





5.00/5 (4 votes)
Further promotion of integration ideas
- Download source code - 3.7 MB
- Download Containers.zip 362.7 KB
- Download "Cube" sample 3.6 MB
- Download sample with processing of one image - 1.5 MB
- Download sample with processing of two images - 3.6 MB
1. Introduction
This article promotes integration idea as well as a lot of my other articles. This idea is illustrated by following problem. We would like visualize 3D object which has texture obtained by digital image processing.
2. Data exchange architecture
2.1 Basic interfaces
All problems of article's title need data exchange. Data exchange architecture has four basic interfaces. First interface is elementary unit of data exchange.
/// <summary> /// Elementary unit of data exchange /// </summary> public interface IMeasure { /// <summary> /// Function which returns unit of data /// </summary> Func<object> Parameter { get; } /// <summary> /// The name of data unit /// </summary> string Name { get; } /// <summary> /// Type of parameter /// </summary> object Type { get; } }
Members of this interface are presented in following table.
N | Member name | Meaning |
1 | Parameter |
Function which returns unit of data |
2 | Name |
The name of data unit |
3 | Type |
Type of parameter |
The Parameter
function can return any object. The Name
property enables us to link input parameters to output ones. The Type
property contains Meta information about unit. This property is not necessary variable of System.Type
type.
Following table is a mapping between variables and their "types".
N | Type of variable | Value of Type property |
1 | string | string s = "" |
2 | bool | bool b = false |
3 | double | double d = 0 |
4 | float | float f = 0 |
5 | byte | byte b = 0x0 |
6 | sbyte | sbyte s = 0x0 |
7 | short | short s = 0 |
8 | short | ushort u = 0 |
9 | int | int i = 0 |
10 | uint | uint u = 0 |
11 | long | long l = 0 |
12 | ulong | ulong u = 0 |
13 | Array | ArrayReturnType a = (depends on type and length of array) |
If elementary unit parameter is array then "type" of unit is ArrayReturnType
variable. The lArrayReturnType
code is presented below.
/// <summary> /// Return type of array /// </summary> public class ArrayReturnType { #region Fields object elementType; int[] dimension; bool isObjectType; #endregion /// <summary> /// Constructor /// </summary> /// <param name="elementType">Type of element</param> /// <param name="dimension">Dimension</param> /// <param name="objectType">The "is object" sign</param> public ArrayReturnType(object elementType, int[] dimension, bool objectType) { this.elementType = elementType; this.dimension = dimension; this.isObjectType = objectType; } /// <summary> /// The "is object" sign /// </summary> public bool IsObjectType { get { return isObjectType; } } /// <summary> /// Dimension /// </summary> public int[] Dimension { get { return dimension; } } /// <summary> /// Type of element /// </summary> public object ElementType { get { return elementType; } } /// <summary> /// Overriden Equals /// </summary> /// <param name="obj">Compared obje</param> /// <returns>True if equal</returns> public override bool Equals(object obj) { if (!(obj is ArrayReturnType)) { return false; } ArrayReturnType at = obj as ArrayReturnType; if (!at.elementType.Equals(elementType)) { return false; } if (at.dimension.Length != dimension.Length) { return false; } for (int i = 0; i < dimension.Length; i++) { if (dimension[i] != at.dimension[i]) { return false; } } return true; } /// <summary> /// Overriden /// </summary> /// <returns>Hash code</returns> public override int GetHashCode() { return dimension.Length * dimension[0]; } /// <summary> /// Checks equality of imension /// </summary> /// <param name="type">Type</param> /// <returns>True in case of equal dimesion and false otherwise</returns> public bool HasEqualDimension(ArrayReturnType type) { if (type.dimension.Length != dimension.Length) { return false; } for (int i = 0; i < dimension.Length; i++) { if (dimension[i] != type.dimension[i]) { return false; } } return true; } /// <summary> /// Gets base type /// </summary> /// <param name="o">The object</param> /// <returns>Object's base type</returns> static public object GetBaseType(object o) { if (o is ArrayReturnType) { ArrayReturnType rt = o as ArrayReturnType; return rt.elementType; } return o; } }
Following table contains samples of variables with their Type
property.
N | Variable | Value of Type property | Comment |
1 | double[] d = new double[3]; | new ArrayReturnType((double)0, new int[]{3}, false); | |
2 | object[] o = new object[2]; | new ArrayReturnType((double)0, new int[]{2}, true); | Every element of o is double |
3 | bool[,] b = new object[2,3]; | new ArrayReturnType(false, new int[]{2, 3}, false); | |
4 | string[,,] s = new string[5,2,3]; | new ArrayReturnType("", new int[]{5, 2, 3}, false); |
Set of supported types of parameters can be extended.
Second interface is implemented by all providers of data.
/// <summary> /// Data provider /// </summary> public interface IMeasurements { /// <summary> /// The count of data units /// </summary> int Count { get; } /// <summary> /// Gets number - th unit of data /// </summary> IMeasure this[int number] { get; } /// <summary> /// Updates data /// </summary> void UpdateMeasurements(); /// <summary> /// Shows, weather the object is updated /// </summary> bool IsUpdated { get; set; } }
Following table explains meaning of this interface members
N | Member name | Meaning | Comment |
1 | Count | The count of data units | |
2 | this[int number] | Gets numberth unit of data | |
3 | UpdateMeasurements() | Updates data | Data can be updated by the time |
4 | IsUpdated | Shows, weather the object is updated | This property is used for avoiding multiple update |
All consumers of data implement following interface.
/// <summary> /// Consumer of data /// </summary> public interface IDataConsumer { /// <summary> /// Adds data provider /// </summary> /// <param name="measurements">Provider to add</param> void Add(IMeasurements measurements); /// <summary> /// Removes data provider /// </summary> /// <param name="measurements">Provider to remove</param> void Remove(IMeasurements measurements); /// <summary> /// Updates data of data providers /// </summary> void UpdateChildrenData(); /// <summary> /// Count of providers /// </summary> int Count { get; } /// <summary> /// Access to n - th provider /// </summary> IMeasurements this[int number] { get; } /// <summary> /// Resets measurements /// </summary> void Reset(); /// <summary> /// Change Input event /// </summary> event Action OnChangeInput; }
Following interface is auxiliary.
/// <summary> /// Collection on named data units /// </summary> public interface IAlias : IAliasBase { /// <summary> /// Names of all data units /// </summary> IList<string> AliasNames { get; } /// <summary> /// Access to data unit by name /// </summary> object this[string name] { get; set; } /// <summary> /// Gets unit type /// </summary> /// <param name="name">Unit name</param> /// <returns>Type of unit</returns> object GetType(string name); }
Following class links data consumers to data providers.
/// <summary> /// The link between data provider and data consumer /// </summary> [Serializable()] public class DataLink : ICategoryArrow, ISerializable, IRemovableObject, IDataLinkFactory { #region Fields /// <summary> /// Error message /// </summary> public static readonly string SetProviderBefore = "You should create measurements source before consumer"; /// <summary> /// DataLink checker /// </summary> private static Action<DataLink> checker; /// <summary> /// The source of this arrow /// </summary> private IDataConsumer source; /// <summary> /// The target of this arrow /// </summary> private IMeasurements target; /// <summary> /// Auxiliary field /// </summary> private int a = 0; /// <summary> /// Linked object /// </summary> protected object obj; /// <summary> /// Data link factory /// </summary> private static IDataLinkFactory dataLinkFactory = new DataLink(); #endregion #region Ctor /// <summary> /// Default constructor /// </summary> public DataLink() { } /// <summary> /// Deserialization constructor /// </summary> /// <param name="info">Serialization info</param> /// <param name="context">Streaming context</param> public DataLink(SerializationInfo info, StreamingContext context) { a = (int)info.GetValue("A", typeof(int)); } #endregion #region ISerializable Members /// <summary> /// ISerializable interface implementation /// </summary> /// <param name="info">Serialization info</param> /// <param name="context">Streaming context</param> public void GetObjectData(SerializationInfo info, StreamingContext context) { } #endregion #region ICategoryArrow Members /// <summary> /// The source of this arrow /// </summary> public ICategoryObject Source { set { if (source != null) { throw new Exception(); } IDataLinkFactory f = this; source = f.GetConsumer(value); } get { return source as ICategoryObject; } } /// <summary> /// The target of this arrow /// </summary> public ICategoryObject Target { get { return target as ICategoryObject; } set { if (target != null) { throw new Exception(); } IDataLinkFactory f = this; bool check = true; IAssociatedObject s = source as IAssociatedObject; if (s.Object != null & value.Object != null) { if (check) { INamedComponent ns = s.Object as INamedComponent; INamedComponent nt = value.Object as INamedComponent; if (nt != null & ns != null) { if (PureDesktopPeer.GetDifference(nt, ns) >= 0) { throw new Exception(SetProviderBefore); } } } target = t; source.Add(target); } if (!check) { return; } try { if (checker != null) { checker(this); } } catch (Exception e) { e.ShowError(10); source.Remove(target); throw e; } } } /// <summary> /// The "is monomorhpism" sign /// </summary> public bool IsMonomorphism { get { return false; } } /// <summary> /// The "is epimorhpism" sign /// </summary> public bool IsEpimorphism { get { return false; } } /// <summary> /// The "is isomorhpism" sign /// </summary> public bool IsIsomorphism { get { return false; } } /// <summary> /// Composes this arrow "f" with next arrow "g" /// </summary> /// <param name="category"> The category of arrow</param> /// <param name="next"> The next arrow "g" </param> /// <returns>Composition "fg" </returns> public ICategoryArrow Compose(ICategory category, ICategoryArrow next) { return null; } #endregion #region IAssociatedObject Members /// <summary> /// Associated object /// </summary> public object Object { get { return obj; } set { obj = value; } } #endregion #region IRemovableObject Members /// <summary> /// The post remove operation /// </summary> public void RemoveObject() { if (source == null | target == null) { return; } source.Remove(target); } #endregion #region IDataLinkFactory Members IDataConsumer IDataLinkFactory.GetConsumer(ICategoryObject source) { IAssociatedObject ao = source; object o = ao.Object; if (o is INamedComponent) { IDataConsumer dcl = null; INamedComponent comp = o as INamedComponent; IDesktop desktop = comp.Root.Desktop; desktop.ForEach<DataLink>(delegate(DataLink dl) { if (dcl != null) { return; } object dt = dl.Source; if (dt is IAssociatedObject) { IAssociatedObject aot = dt as IAssociatedObject; if (aot.Object == o) { dcl = dl.source as IDataConsumer; } } }); if (dcl != null) { return dcl; } } IDataConsumer dc = DataConsumerWrapper.Create(source); if (dc == null) { CategoryException.ThrowIllegalTargetException(); } return dc; } IMeasurements IDataLinkFactory.GetMeasurements(ICategoryObject target) { IAssociatedObject ao = target; object o = ao.Object; if (o is INamedComponent) { IMeasurements ml = null; INamedComponent comp = o as INamedComponent; IDesktop d = null; INamedComponent r = comp.Root; if (r != null) { d = r.Desktop; } else { d = comp.Desktop; } if (d != null) { d.ForEach<DataLink>(delegate(DataLink dl) { if (ml != null) { return; } object dt = dl.Target; if (dt is IAssociatedObject) { IAssociatedObject aot = dt as IAssociatedObject; if (aot.Object == o) { ml = dl.Target as IMeasurements; } } }); if (ml != null) { return ml; } } } IMeasurements m = MeasurementsWrapper.Create(target); if (m == null) { CategoryException.ThrowIllegalTargetException(); } return m; } #endregion #region Public Members /// <summary> /// Checker of data link /// </summary> public static Action<DataLink> Checker { set { checker = value; } } /// <summary> /// Data link factory /// </summary> public static IDataLinkFactory DataLinkFactory { get { return dataLinkFactory; } set { dataLinkFactory = value; } } /// <summary> /// Measurements provider /// </summary> public IMeasurements Measurements { get { return target; } } #endregion }
implies execution of following operator
consumer.Add(provider); // Consumer adds a provider.
Indeed above operators are never written explicitly, but they are implied by following graphical designer.
The Link arrow corresponds to DataLink
object. Source (Chart) (resp. target (Formula)) object is object which implements IDataConsumer
(resp. IMeasurements
) interface. Graphical setting of Link arrow
implies execution of next operator:
consumer.Add(provider); // Consumer adds a provider
Deleting of the arrow implies execution of next operator:
consumer.Remove(provider); // Consumer removes a provider
2.2 Sample
Following sample exhibits data exchange.
This sample contains three objects and three DataLink
arrows. Following table contains types of objects.
N | Object | Type | Implemented interfaces |
1 | Formula | VectorFormulaConsumer | IDataConsumer , IMeasurements , IAlias |
2 | Differential equation | DifferentialEquationSolver | IDataConsumer , IMeasurements , IAlias |
3 | Chart | DataConsumer | IDataConsumer |
VectorFormulaConsumer
and DifferentialEquationSolver
implement both IDataConsumer
and IMeasurements
interfaces. So these objects can be both sources and targets of DataLink
arrow. But DataConsumer
object can be source only.
The Formula object has following properties
Variables a and b are marked. It means that these variables correspond to IAlias
interface. Following code snippet explains this circumstance.
VectorFormulaConsumer formula = ...; IAlias alias = formula; IList<string> names = alias.AliasNames; // names[0] = "a", names[1] = "b"; double a = (double)alias["a"]; // a = 1; double b = (double)alias["b"]; // b = 2; object ta = alias.GetType("a"); // ta = (double)0; object tb = alias.GetType("b"); // tb = (double)0;
The t variable corresponds to virtual time.
The Differential equation object has following properties
This object performs solution of following system of ordinary differential equations
The Chart object indicates time dependencies. Its properties are clear.
3. Kinematics architecture
3.1 Main idea
Frame of reference is a basic notion of kinematics. Presented here architecture uses relative frames of reference, those have forest structure. Example of such structure is presented in the following diagram:
There exists a zero frame of reference. All root nodes of the forest correspond to this frame. 6D positions of other frames (nodes) are relative to 6D positions of those parents. It means that 6D position of frame 1.1.2 is relative to 6D position of frame 1.1 etc. This architecture enables us easily simulate a lot of different useful situations. For example we can install a set of virtual cameras on air(space)craft. Also it is easy to visualize complicated motion of helicopter rotor (slow motion) a plane with swing-wing or thrust vector control.
3.2 Basic classes and interfaces
Following two interfaces correspond to 3D position and 3D orientation respectively
/// <summary> /// 3D Position /// </summary> public interface IPosition { /// <summary> /// Absolute position coordinates /// </summary> double[] Position { get; } /// <summary> /// Parent frame /// </summary> IReferenceFrame Parent { get; set; } /// <summary> /// Position parameters /// </summary> object Parameters { get; set; } /// <summary> /// Updates itself /// </summary> void Update(); }
/// <summary> /// Object with orientation /// </summary> public interface IOrientation { /// <summary> /// Orientation quaternion /// </summary> double[] Quaternion { get; } /// <summary> /// Orientation matrix /// </summary> double[,] Matrix { get; } }
The ReferenceFrame
class implements both IPosition
and IOrientation
. Its code is very long and is contained is source files. Following interface supplies relative motion paradigm.
/// <summary> /// Reference frame holder /// </summary> public interface IReferenceFrame : IPosition { /// <summary> /// Own frame /// </summary> ReferenceFrame Own { get; } /// <summary> /// Children objects /// </summary> List<IPosition> Children { get; } }
The Parent
property of IReferenceFrame
interface is the parent frame of reference. Motion of point is relative with respect to parent frame. The Children
property of IReferenceFrame
is collection of children points.
Following class links child to parent.
/// <summary> /// Link of relative frame /// </summary> [Serializable()] public class ReferenceFrameArrow : CategoryArrow, ISerializable, IRemovableObject { #region Fields IPosition source; IReferenceFrame target; #endregion #region Constructors /// <summary> /// Default constructor /// </summary> public ReferenceFrameArrow() { } /// <summary> /// Deserialization constructor /// </summary> /// <param name="info">Serialization info</param> /// <param name="context">Streaming context</param> protected ReferenceFrameArrow(SerializationInfo info, StreamingContext context) { } #endregion #region ISerializable Members void ISerializable.GetObjectData(SerializationInfo info, StreamingContext context) { } #endregion #region ICategoryArrow Members /// <summary> /// The source of this arrow /// </summary> public override ICategoryObject Source { get { return source as ICategoryObject; } set { IPosition position = value.GetSource<IPosition>(); if (position.Parent != null) { throw new CategoryException("Root", this); } source = position; } } /// <summary> /// The target of this arrow /// </summary> public override ICategoryObject Target { get { return target as ICategoryObject; } set { IReferenceFrame rf = value.GetTarget<IReferenceFrame>(); IAssociatedObject sa = source as IAssociatedObject; IAssociatedObject ta = value as IAssociatedObject; INamedComponent ns = sa.Object as INamedComponent; INamedComponent nt = ta.Object as INamedComponent; target = rf; source.Parent = target; target.Children.Add(source); } } #endregion #region IRemovableObject Members void IRemovableObject.RemoveObject() { source.Parent = null; if (target != null) { target.Children.Remove(source); } } #endregion #region Specific Members /// <summary> /// Preparation operation /// </summary> /// <param name="collection">Desktop</param> /// <returns>List of position objects</returns> static public List<IPosition> Prepare(IComponentCollection collection) { List<IPosition> frames = new List<IPosition>(); if (collection == null) { return frames; } IEnumerable<object> c = collection.AllComponents; foreach (object o in c) { if (!(o is IObjectLabel)) { continue; } IObjectLabel lab = o as IObjectLabel; ICategoryObject co = lab.Object; if (!(co is IReferenceFrame)) { if (co is IPosition) { IPosition p = co as IPosition; if (p.Parent == null) { frames.Add(p); } } continue; } IReferenceFrame f = co as IReferenceFrame; if (f.Parent != null) { continue; } prepare(f, frames); } return frames; } /// <summary> /// Updates frames /// </summary> /// <param name="frames">List of frames</param> public static void Update(List<IPosition> frames) { foreach (IPosition frame in frames) { frame.Update(); } } private static void prepare(IReferenceFrame frame, List<IPosition> frames) { List<IPosition> children = frame.Children; frames.Add(frame); foreach (IPosition p in children) { if (frames.Contains(p)) { continue; } if (p is IReferenceFrame) { IReferenceFrame f = p as IReferenceFrame; prepare(f, frames); } else { frames.Add(p); } } } #endregion }
Source of this arrow is IPoint
object, target is a IReferenceFrame
one.
Setting of ReferenceFrameArrow
arrow
IPosition position = ...; IReferenceFrame frame = ...; ReferenceFrameArrow link = new ReferenceFrameArrow(); link.Source = position as ICategoryObject; link.Target = position as ICategoryObject;
implies execution of following operators
position.Parent = frame; frame.Children.Add(position);.
Setting of all arrows is provided by graphical designer.
This picture has three links L1, L2 and L3 of ReferenceFrameArrow
type and four objects
N | Name | Type | Implemented interfaces | Parent |
1 | Base frame | RigidReferenceFrame |
IPosition , IReferenceFrame | null |
2 | Frame 1 | ReferenceFrameData |
IPosition , IReferenceFrame , IDataConsumer | Base frame |
3 | Frame 2 | ReferenceFrameData |
IPosition , IReferenceFrame , IDataConsumer | Frame 1 |
4 | Camera frame | RigidReferenceFrame |
IPosition , IReferenceFrame | Frame 2 |
5 | Camera | WpfCamera |
IPosition | Camera frame |
Since parent of Base frame is null
motion of this frame is regarded as motion with respect to zero frame of reference. The RigidReferenceFrame
class corresponds to frames with constant relative coordinates and orientation. The ReferenceFrameData
class corresponds to moving frames of reference. This class implements IDataConsumer
interface for interoperability with data exchange.
3.3 Interoperability with data exchange
The ReferenceFrameData
implements IDataConsumer
interface. It uses external data as parameters of relative motions. Following picture represents application of ReferenceFrameData
The Motion Parameters object calculates motion parameters by following way:
This object as IMeasurements
is linked to Motion Frame as IDataConsumer
by DataLink
D. The Motion Frame object uses output data of Motion Parameters.
N | Motion parameter of Motion Frame | Output parameter of Motion Parameters |
1 | Relative x - coordinate | Formula_3 |
2 | Relative y - coordinate | Formula_4 |
3 | Relative z - coordinate | Formula_4 |
4 | Q0 component of relative orientation quaternion | Formula_1 |
5 | Q1 component of relative orientation quaternion | Formula_2 |
6 | Q2 component of relative orientation quaternion | Formula_4 |
7 | Q3 component of relative orientation quaternion | Formula_4 |
Besides IDataConsumer kinematic architecture contains RelativeMeasurements
which implements IMeasurements
interface.
Type of above Relative object is RelativeMeasurements
. The R1, R2 arrow between Relative and frames mean that Relative provides parameters of Motion Frame relative motion with respect to Base frame. Since Relative implements IMeasurements
interface it can be connected by DataLink
to IDataConsumer
Here Chart object indicates relative x, y and z coordinates.
4. Digital image processing architecture
4.1 Main interfaces and classes
Roughly speaking digital image processing obtains one image from other ones. So architecture contains provides and consumers of images. Following interface is provider of images.
/// <summary> /// Provider of image /// </summary> public interface IBitmapProvider { /// <summary> /// Bitmap /// </summary> Bitmap Bitmap { get; } }
Following interface is consumer of image.
/// <summary> /// Consumer of image /// </summary> public interface IBitmapConsumer { /// <summary> /// Procesess image /// </summary> void Process(); /// <summary> /// Providers /// </summary> IEnumerable<IBitmapProvider> Providers { get; } /// <summary> /// Adds a provider /// </summary> /// <param name="provider">The provider</param> void Add(IBitmapProvider provider); /// <summary> /// Removes a provider /// </summary> /// <param name="provider">The provider</param> void Remove(IBitmapProvider provider); }
Following class is link between IBitmapProvider
and IBitmapConsumer
/// <summary> /// Link between bitmap consumer and bitmap provider /// </summary> [Serializable()] public class BitmapConsumerLink : ICategoryArrow, IRemovableObject, ISerializable { #region Fields /// <summary> /// Error message /// </summary> static public readonly string ProviderExists = "Bitmap provider already exists"; /// <summary> /// Error message /// </summary> public static readonly string SetProviderBefore = "You should create bitmap provider before consumer"; /// <summary> /// Associated object /// </summary> private object obj; /// <summary> /// Auxiliary variable /// </summary> private int a = 0; /// <summary> /// Source /// </summary> private IBitmapConsumer source; /// <summary> /// Target /// </summary> private IBitmapProvider target; #endregion #region Constructors public BitmapConsumerLink() { } public BitmapConsumerLink(SerializationInfo info, StreamingContext context) { info.GetValue("A", typeof(int)); } #endregion #region ICategoryArrow Members public ICategoryObject Source { get { return source as ICategoryObject; } set { source = value.GetObject<IBitmapConsumer>(); } } public ICategoryObject Target { get { return target as ICategoryObject; } set { target = value.GetObject<IBitmapProvider>(); source.Add(target); } } public bool IsMonomorphism { get { return false; } } public bool IsEpimorphism { get { return false; } } public bool IsIsomorphism { get { return false; } } public ICategoryArrow Compose(ICategory category, ICategoryArrow next) { return null; } #endregion #region IAssociatedObject Members public object Object { get { return obj; } set { obj = value; } } #endregion #region IRemovableObject Members public void RemoveObject() { if (source != null & target != null) { source.Remove(target); } } #endregion #region ISerializable Members public void GetObjectData(SerializationInfo info, StreamingContext context) { } #endregion #region Specific Members /// <summary> /// Updates consumer /// </summary> /// <param name="consumer">Consumer</param> public static void Update(IBitmapConsumer consumer) { IEnumerable<IBitmapProvider> providers = consumer.Providers; foreach (IBitmapProvider provider in providers) { if (provider is IBitmapConsumer) { IBitmapConsumer c = provider as IBitmapConsumer; Update(c); } } consumer.Process(); } /// <summary> /// Gets provider for consumer /// </summary> /// <param name="provider">Provider</param> /// <param name="consumer">Consumer</param> /// <param name="mutipleProviders">The multiple providers flag</param> /// <returns>The provider</returns> public static IBitmapProvider GetProvider(IBitmapProvider provider, IBitmapConsumer consumer, bool mutipleProviders) { if (provider == null) { return null; } ICategoryObject t = provider as ICategoryObject; ICategoryObject s = consumer as ICategoryObject; if (s.Object != null & t.Object != null) { INamedComponent ns = s.Object as INamedComponent; INamedComponent nt = t.Object as INamedComponent; if (nt != null & ns != null) { if (nt.Desktop == ns.Desktop) { if (nt.Ord >= ns.Ord) { throw new Exception(SetProviderBefore); } } else { if (nt.Root.Ord >= ns.Root.Ord) { throw new Exception(SetProviderBefore); } } } } return provider; } #endregion }
Setting of BitmapConsumerLink
arrow
IBitmapProvider provider = ...; IBitmapConsumer consumer = ...; BitmapConsumerLink link = new BitmapConsumerLink(); link.Source = provider as ICategoryObject; link.Target = consumer as ICategoryObject;
implies execution of following operator
consumer.Add(provider);. However above operations are performed by graphics designer.
Here Lady Rose and Lady Blue are IBitmapProvider
and IBitmapConsumer
respectively. These objects are linked by BitmapConsumerLink
4.2 Interoperability with data exchange
4.2.1 Processing of one bitmap
Digital image processing has classes which implement IDataConsumer
and/or IMeasurements
interfaces. Let us consider local digital filtration sample.
N | Object name | Type | Implemented interfaces |
1 | Earth | SourceBitmap | IBitmapProvider |
2 | Result of processing | BitmapTransformer | IBitmapProvider , IBitmapConsumer , IDataConsumer |
3 | Formulae | VectorFormulaConsumer | IMeasurements , IAlias |
The Result of processing object as IBitmapConsumer
is connected to Earth as IBitmapProvider
by BitmapConsumerLink
. The Result of processing object as IDataConsumer
is connected to Formulae as IMeasurements
by DataLink
. Any object of SourceBitmap
class stores just stores image in memory. Main members of this class are presented below.
#region Fields /// <summary> /// Bitmap /// </summary> protected Bitmap bitmap; #endregion #region Ctor /// <summary> /// Default constructor /// </summary> public SourceBitmap() { } /// <summary> /// Deserialization constructor /// </summary> /// <param name="info">Serialization info</param> /// <param name="context">Streaming context</param> public SourceBitmap(SerializationInfo info, StreamingContext context) { try { bitmap = (Bitmap)info.GetValue("Bitmap", typeof(Bitmap)); } catch (Exception ex) { ex.ShowError(100); } } #endregion #region ISerializable Members /// <summary> /// ISerializable interface implementation /// </summary> /// <param name="info">Serialization info</param> /// <param name="context">Streaming context</param> public override void GetObjectData(SerializationInfo info, StreamingContext context) { info.AddValue("Bitmap", bitmap); } #region IBitmapProvider Members /// <summary> /// Bitmap /// </summary> Bitmap IBitmapProvider.Bitmap { get { return bitmap; } } #endregion /// <summary> /// Sets bitmap /// </summary> /// <param name="bitmap"></param> public void SetBitmap(Bitmap bitmap) { this.bitmap = bitmap; } // Other members ...
Business logic of SourceBitmap
is very clear. This class has a field of Bitmap
type. User can set value of this field? This field is serialized and is also used as Bitmap
of IBitmapProvider
interface. The Formulae object has following properties.
This formula calculates color from color of source bitmap. Parameters r, g and b correspond to red, green and blue color of source bitmap. If values of all colors exceed threshold value (a then formula returns x. Otherwise it returns y. Object Result of processing following properties.
These properties have following meaning. Object scans bitmap of provider object (Earth) and detects colors of pixels. It sets these objects as aliases of Formulae. Then it calculates Formulae and sets Formula_1 as red, green and blue color of result bitmap. Source bitmap and transformation result are presented below.
This algorithm is simplest rough algorithm of detection of snow mantle. C# explanation of this algorithm is presented below.
//============ Formulae ============== VectorFormulaConsumer formulae = ...; // Formulae object IMeasurements measurements = formulae; IMeasure m = measurements[0]; // Formula_1 IAlias alias = formulae; // Formulae object as IAlias // ========================================= // =========== Source bitmap =============== SourceBitmap sb = ...; IBitmapProvider provider = sb; Bitmap source = provider.Bitmap; //=========================================== // ============ Target bitmap =============== Bitmap target = // Size of tagtet is equal to source one new Bitmap(source.Width, source.Height); for (int x = 0; x < source.Width; x++) // x = coordinate of bitmap { for (int y = 0; y < source.Height; y++) // y - coordinate of bitmap { Color colorSource = source.GetPixel(x, y); // Color of source pixel double r = (double)colorSource.R / 256; // Scaling double g = (double)colorSource.G / 256; double b = (double)colorSource.B / 256; alias["r"] = r; // Setting of aliases alias["g"] = g; alias["b"] = b; measurements.UpdateMeasurements(); // Calculates all formulae double cd = (double)m.Parameter(); // Formula_1 int cdi = (int)(cd * 256); // Inverse scaling Color colorTarget = Color.FromArgb(cdi, cdi, cdi); // Target color target.SetPixel(x, y, colorTarget); // Sets pixel to target bitmap } }
Note that colors of target can be different as it is shown in "Lady Blue" sample. Above code snippet does not present in my code, it is just clear explanation of algorithm. More information about this subject you can find in my article "Digital Image Processing".
4.2.2 Processing of two bitmaps
Several tasks imply simultaneous processing of several images. For example clouds dynamics indication requires comparison of two or more images. Following picture represents comparison of two images
This picture has following objects
N | Object name | Type | Implemented interfaces | Comment |
1 | Picture 1 | SourceBitmap | IBitmapProvider | First source image |
2 | Picture 2 | SourceBitmap | IBitmapProvider | Second source image |
3 | P 1 | BitmapColorTable | IBitmapConsumer , IDataConsumer , IMeasurements | Adapter object |
4 | P 2 | BitmapColorTable | IBitmapConsumer , IDataConsumer , IMeasurements | Adapter object |
5 | Input | VectorFormulaConsumer | IMeasurements , IAlias | Digital image processing calculator |
6 | Result | VectorFormulaConsumer | IMeasurements , IAlias | Digital image processing calculator |
7 | Compare | BitmapTransformer | IBitmapProvider , IBitmapConsumer , IDataConsumer | Digital image processing result |
The Compare object as IBitmapConsumer
is connected to Picture 2 as IBitmapProvider
by BitmapConsumerLink
. It means that P 2 provides bitmap for Compare. According to BitmapTransformer
implementation it means that size of Compare bitmap is the same as Picture 2 one. The Compare has following properties.
These properties have following meaning. The Compare object scans own bitmap and sets alias parameters as values of pixel coordinates. Then it sets Formula_1 value as red, green and blue color of image. Following code clarifies this algorithm.
VectorFormulaConsumer Input = ...; // The "Input" object VectorFormulaConsumer Result = ...; // The "Result" object IAlias alias = Input; // "Input" as IAlias IMeasurements measurements = // "Result" as IMeasurements Result; IMeasure Formula_1 = measurements[0]; // "Formula_1" Bitmap Compare = ...; // Bitmap of "Compare object" for (int x = 0; x < Compare.Width; x++) // x = coordinate of bitmap { for (int y = 0; y < Compare.Height; y++) // y - coordinate of bitmap { alias["x"] = (double)x; // Setting parameters alias["y"] = (double)y; measurements.UpdateMeasurements(); // Calculation int color = // Color (int)((double)Formula_1.Parameter() * 256); Color c = Color.FromArgb(color, color, color); Compare.SetPixel(x, y, c); // Sets pixel color } }
Both P 1 and P 2 are objects of BitmapColorTable
type. This type as IDataConsumer
consumes coordinates pixel, and as IMeasurements
provides RGB color parameters. Following picture
means that P 1 returns RGB parameters of Picture 1 bitmap pixel. Coordinate x (resp. y) of pixel equals to Formula_1, (resp. Formula_2) of Input. Properties of Input are presented below.
So parameter x (resp. y) of Input as IAlias
equals to Formula_1 (resp. Formula_2) of the same object as IMeasurements
. Thus input parameters of both P 1, P 2 are pixel coordinates of Compare bitmap. Properties of Result are presented below.
Parameter x (resp. y) of this object is Red parameter of P 1 (resp. P 2), i.e. red component of pixel of Picture 1 (resp. Picture 2). Formula_1 of Result is proportional to difference between red components of Picture 1 and Picture 2.
5. Architecture of 3D graphics
5.1 Main interfaces and classes
Main interfaces of 3D graphics are IVisible
and IVisibleConsumer
/// <summary> /// Object linked to position /// </summary> public interface IPositionObject { /// <summary> /// Linked position /// </summary> IPosition Position { get; set; } } /// <summary> /// Visible 3D object /// </summary> public interface IVisible : IPositionObject { } /// <summary> /// Consumer of visible 3D object /// </summary> public interface IVisibleConsumer { /// <summary> /// Adds visible object to consumer /// </summary> /// <param name="visible">Visible object to add</param> void Add(IVisible visible); /// <summary> /// Removes visible object from consumer /// </summary> /// <param name="visible">Visible object to remove</param> void Remove(IVisible visible); /// <summary> /// Post operation /// </summary> /// <param name="visible">Visible object</param> void Post(IVisible visible); }
Class VisibleConsumerLink
links IVisibleConsumer
to IVisible
. Setting of VisibleConsumerLink
arrow
IVisible visible = ...; IVisibleConsumer consumer = ...; ICategoryArrow link = new VisibleConsumerLink(); link.Source = consumer as ICategoryObject; link.Target = visible as ICategoryObject;
implies execution of following operator
consumer.Add(visible);
Removing of VisibleConsumerLink
follows that
#region IRemovableObject Members /// <summary> /// Removing of VisibleConsumerLink /// </summary> void IRemovableObject.RemoveObject() { source.Remove(target); } #endregion
Following picture represents a sample of these interfaces.
Objects and arrows of above pictures have following types
N | Object name | Type | Implemented interfaces | Comment |
1 | Camera | WpfCamera | IVisibleConsumer | Wrapper of PerspectiveCamera |
2 | Globe | WpfShape | IVisible | Wrapper of Visual |
3 | Link | VisibleConsumerLink | ICategoryArrow | Links IVisibleConsumer with IVisible object |
Above example means that Camera consumes Globe, and "consumes" means that visualizes.
5.2 Universality of architecture
This architecture operates with interfaces which do not know about WPF. So this software can be adapted to other 3D graphics technologies, for example OpenGL. I have already described compatibility of this software with different 3D technologies in "The time machine" article.
5.3 Interoperability with kinematics
Any object of 3D interface is associated with frame of reference. Next sample contains 3D object (Cube) with textures and four virtual cameras (Perspective Camera, X - camera, Y - camera and Z - camera).
The cube is linked by L 1 to moving frame of reference Frame 3 cameras are linked to fixed frames of reference. So cameras indicate Cube motion from different 3D points.
5.4 Features of WPF implementation of 3D graphics.
WPF implementation uses XAML file and texture files.
Following code is XAML file content
<ModelVisual3D xmlns="http://schemas.microsoft.com/winfx/2006/xaml/presentation"> <ModelVisual3D.Children> <ModelVisual3D> <ModelVisual3D.Content> <AmbientLight Color="#333333" /> </ModelVisual3D.Content> </ModelVisual3D> <ModelVisual3D> <ModelVisual3D.Content> <DirectionalLight Color="#FFFFFF" Direction="-0.612372,-0.5,-0.612372" /> </ModelVisual3D.Content> </ModelVisual3D> <ModelVisual3D> <ModelVisual3D.Content> <DirectionalLight Color="#FFFFFF" Direction="0.612372,-0.5,-0.612372" /> </ModelVisual3D.Content> </ModelVisual3D> <ModelVisual3D> <ModelVisual3D.Content> <GeometryModel3D> <GeometryModel3D.Material> <MaterialGroup> <DiffuseMaterial> <DiffuseMaterial.Brush> <ImageBrush Stretch="UniformToFill" ImageSource="leaves_closeup.png" TileMode="None" ViewportUnits="Absolute" Viewport="0 0 1 1" AlignmentX="Left" AlignmentY="Top" Opacity="1.000000" /> </DiffuseMaterial.Brush> </DiffuseMaterial> <SpecularMaterial SpecularPower="85.3333"> <SpecularMaterial.Brush> <SolidColorBrush Color="#FFFFFF" Opacity="1.000000"/> </SpecularMaterial.Brush> </SpecularMaterial> </MaterialGroup> </GeometryModel3D.Material> <GeometryModel3D.Geometry> <MeshGeometry3D TriangleIndices="0,1,2 3,4,5" Normals="-1,0,0 -1,0,0 -1,0,0 -1,0,0 -1,0,0 -1,0,0 " TextureCoordinates="0,1 0,0 1,0 1,0 1,1 0,1 " Positions="-0.5,0.5,-0.5 -0.5,-0.5,-0.5 -0.5,-0.5,0.5 -0.5,-0.5,0.5 -0.5,0.5,0.5 -0.5,0.5,-0.5 " /> </GeometryModel3D.Geometry> </GeometryModel3D> </ModelVisual3D.Content> </ModelVisual3D> <ModelVisual3D> <ModelVisual3D.Content> <GeometryModel3D> <GeometryModel3D.Material> <MaterialGroup> <DiffuseMaterial> <DiffuseMaterial.Brush> <ImageBrush Stretch="UniformToFill" ImageSource="rocks.png" TileMode="None" ViewportUnits="Absolute" Viewport="0 0 1 1" AlignmentX="Left" AlignmentY="Top" Opacity="1.000000" /> </DiffuseMaterial.Brush> </DiffuseMaterial> <SpecularMaterial SpecularPower="85.3333"> <SpecularMaterial.Brush> <SolidColorBrush Color="#FFFFFF" Opacity="1.000000"/> </SpecularMaterial.Brush> </SpecularMaterial> </MaterialGroup> </GeometryModel3D.Material> <GeometryModel3D.Geometry> <MeshGeometry3D TriangleIndices="0,1,2 3,4,5" Normals="0,0,1 0,0,1 0,0,1 0,0,1 0,0,1 0,0,1 " TextureCoordinates="0,0 1,0 1,1 1,1 0,1 0,0 " Positions="-0.5,-0.5,0.5 0.5,-0.5,0.5 0.5,0.5,0.5 0.5,0.5,0.5 -0.5,0.5,0.5 -0.5,-0.5,0.5 " /> </GeometryModel3D.Geometry> </GeometryModel3D> </ModelVisual3D.Content> </ModelVisual3D> <ModelVisual3D> <ModelVisual3D.Content> <GeometryModel3D> <GeometryModel3D.Material> <MaterialGroup> <DiffuseMaterial> <DiffuseMaterial.Brush> <ImageBrush Stretch="UniformToFill" ImageSource= "branches.png" TileMode="None" ViewportUnits="Absolute" Viewport="0 0 1 1" AlignmentX="Left" AlignmentY="Top" Opacity="1.000000" /> </DiffuseMaterial.Brush> </DiffuseMaterial> <SpecularMaterial SpecularPower="85.3333"> <SpecularMaterial.Brush> <SolidColorBrush Color="#FFFFFF" Opacity="1.000000"/> </SpecularMaterial.Brush> </SpecularMaterial> </MaterialGroup> </GeometryModel3D.Material> <GeometryModel3D.Geometry> <MeshGeometry3D TriangleIndices="0,1,2 3,4,5" Normals="0,0,1 0,0,1 0,0,1 0,0,1 0,0,1 0,0,1 " TextureCoordinates="1,0 1,1 0,1 0,1 0,0 1,0 " Positions="0.5,-0.5,-0.5 0.5,0.5,-0.5 0.5,0.5,0.5 0.5,0.5,0.5 0.5,-0.5,0.5 0.5,-0.5,-0.5 " /> </GeometryModel3D.Geometry> </GeometryModel3D> </ModelVisual3D.Content> </ModelVisual3D> <ModelVisual3D> <ModelVisual3D.Content> <GeometryModel3D> <GeometryModel3D.Material> <MaterialGroup> <DiffuseMaterial> <DiffuseMaterial.Brush> <ImageBrush Stretch="UniformToFill" ImageSource="berries.jpg" TileMode="None" ViewportUnits="Absolute" Viewport="0 0 1 1" AlignmentX="Left" AlignmentY="Top" Opacity="1.000000" /> </DiffuseMaterial.Brush> </DiffuseMaterial> <SpecularMaterial SpecularPower="85.3333"> <SpecularMaterial.Brush> <SolidColorBrush Color="#FFFFFF" Opacity="1.000000"/> </SpecularMaterial.Brush> </SpecularMaterial> </MaterialGroup> </GeometryModel3D.Material> <GeometryModel3D.Geometry> <MeshGeometry3D TriangleIndices="0,1,2 3,4,5" Normals="1,0,0 1,0,0 1,0,0 1,0,0 1,0,0 1,0,0 " TextureCoordinates="1,0 1,1 0,1 0,1 0,0 1,0 " Positions="-0.5,-0.5,-0.5 -0.5,0.5,-0.5 0.5,0.5,-0.5 0.5,0.5,-0.5 0.5,-0.5,-0.5 -0.5,-0.5,-0.5 " /> </GeometryModel3D.Geometry> </GeometryModel3D> </ModelVisual3D.Content> </ModelVisual3D> <ModelVisual3D> <ModelVisual3D.Content> <GeometryModel3D> <GeometryModel3D.Material> <MaterialGroup> <DiffuseMaterial> <DiffuseMaterial.Brush> <ImageBrush Stretch="UniformToFill" ImageSource="Waterlilies.png" ViewportUnits="Absolute" Viewport="0 0 1 1" AlignmentX="Left" AlignmentY="Top" Opacity="1.000000" /> </DiffuseMaterial.Brush> </DiffuseMaterial> <SpecularMaterial SpecularPower="85.3333"> <SpecularMaterial.Brush> <SolidColorBrush Color="#FFFFFF" Opacity="1.000000"/> </SpecularMaterial.Brush> </SpecularMaterial> </MaterialGroup> </GeometryModel3D.Material> <GeometryModel3D.Geometry> <MeshGeometry3D TriangleIndices="0,1,2 3,4,5 6,7,8 9,10,11" Normals="0,-1,0 0,-1,0 0,-1,0 0,-1,0 0,-1,0 0,-1,0 0,1,0 0,1,0 0,1,0 0,1,0 0,1,0 0,1,0 " TextureCoordinates="0,0 1,0 1,1 1,1 0,1 0,0 1,1 0,1 0,0 0,0 1,0 1,1 " Positions="-0.5,-0.5,-0.5 -0.5,0.5,0.5 -0.5,-0.5,0.5 0.5,-0.5,0.5 -0.5,-0.5,-0.5 -0.5,0.5,-0.5 0.5,0.5,-0.5 -0.5,0.5,-0.5 -0.5,0.5,0.5 -0.5,0.5,0.5 0.5,0.5,0.5 0.5,0.5,-0.5 " /> </GeometryModel3D.Geometry> </GeometryModel3D> </ModelVisual3D.Content> </ModelVisual3D> <ModelVisual3D > <ModelVisual3D.Content> <GeometryModel3D> <GeometryModel3D.Material> <MaterialGroup> <DiffuseMaterial> <DiffuseMaterial.Brush> <ImageBrush Stretch="UniformToFill" ImageSource="Sunset.jpg" ViewportUnits="Absolute" Viewport="0 0 1 1" AlignmentX="Left" AlignmentY="Top" Opacity="1.000000" /> </DiffuseMaterial.Brush> </DiffuseMaterial> <SpecularMaterial SpecularPower="85.3333"> <SpecularMaterial.Brush> <SolidColorBrush Color="#FFFFFF" Opacity="1.000000"/> </SpecularMaterial.Brush> </SpecularMaterial> </MaterialGroup> </GeometryModel3D.Material> <GeometryModel3D.Geometry> <MeshGeometry3D TriangleIndices="0,1,2 3,4,5 6,7,8 9,10,11" Normals="-1,0,0 -1,0,0 -1,0,0 -1,0,0 -1,0,0 -1,0,0 " TextureCoordinates="1,0 1,1 0,1 0,1 0,0 1,0 " Positions="-0.5,-0.5,0.5 -0.5,-0.5,-0.5 0.5,-0.5,-0.5 0.5,-0.5,-0.5 0.5,-0.5,0.5 -0.5,-0.5,0.5" /> </GeometryModel3D.Geometry> </GeometryModel3D> </ModelVisual3D.Content> </ModelVisual3D> </ModelVisual3D.Children> <ModelVisual3D.Transform> <Transform3DGroup > <Transform3DGroup.Children> <RotateTransform3D> <RotateTransform3D.Rotation> <AxisAngleRotation3D Angle="0" Axis="0 1 0" /> </RotateTransform3D.Rotation> </RotateTransform3D> <RotateTransform3D> <RotateTransform3D.Rotation> <AxisAngleRotation3D Angle="0" Axis="1 0 0" /> </RotateTransform3D.Rotation> </RotateTransform3D> <TranslateTransform3D OffsetX="0" OffsetY="0" OffsetZ="0" /> </Transform3DGroup.Children> </Transform3DGroup> </ModelVisual3D.Transform> </ModelVisual3D>
This content has references to texture files, for example
<ImageBrush Stretch="UniformToFill" ImageSource="berries.jpg" TileMode="None" ViewportUnits="Absolute" Viewport="0 0 1 1" AlignmentX="Left" AlignmentY="Top" Opacity="1.000000" />
WpfShape
class is a wrapper of these objects and it serializes both XAML file and content of texture files.
info.AddValue("Xaml", xaml); info.AddValue("Textures", textures, typeof(Dictionary<string, byte[]>));
Following code explains how textures
dictionary is obtained.
string[] files = new string[] // File names { "berries.jpg", "branches.png", "leaves_closeup.png", "rocks.png", "Sunset.jpg", "Waterlilies.png" }; foreach (string fileName in files) { using (Stream stream = File.OpenRead(fileName)) { byte[] bytes = new byte[stream.Length]; stream.Read(bytes, 0, bytes.Length); // Reads file textures[fileName] = bytes; // Fills texture dictionary } }
6. Digital image processing + 3D graphics
6.1 Main idea
Interoperability between digital image processing and 3D graphics is supported by replacing texture images with digital image processing ones. Following code represents essential features interoperability class.
/// <summary> /// 3D object with digital image processing interoperability /// </summary> [Serializable()] public class MotionImageFigure : WpfShape, IBitmapConsumer, IPostSetArrow { #region Fields /// <summary> /// Textures - Names of providers /// </summary> Dictionary<string, string> dTextures = new Dictionary<string, string>(); /// <summary> /// Textures - Bitmap poviders /// </summary> Dictionary<string, IBitmapProvider> providers = new Dictionary<string, IBitmapProvider>(); /// <summary> /// Post method /// </summary> protected virtual void Post() { foreach (string textureName in dTextures.Keys) // Textures cycle { string providerName = dTextures[textureName]; // Name of provider if (providers.ContainsKey(providerName)) { IBitmapProvider p = providers[providerName]; // Bitmap provider Bitmap bmp = p.Bitmap; // Bitmap of provider paths.Remove(textureName); using (MemoryStream stream = new MemoryStream()) { bmp.Save(stream, System.Drawing.Imaging.ImageFormat.Bmp); // Saving bitmap to bypes textures[textureName] = stream.GetBuffer(); // Replace texture by bitmap } } } }
Above Post()
method yields replacement of texture images by digital image processing ones.
6.2 Elementary sample
Following picture represents usage of MotionImageFigure
.
Type of Cube object is MotionImageFigure
. This object has textures obtained from following graphical files:
- berries.jpg
- branches.png
- leaves_closeup.pn
- rocks.png
- Sunset.jpg
- Waterlilies.png
Some of these textures can be replaced as.
Here textures are replaced by following way.
N | Texture file | Name of provider for replacement |
1 | rocks.png | Compare |
2 | branches.png | Result of processing |
6.3 Advanced sample
In my "Determination of Orbits of Artificial Satellites" article a complicated engineering task is considered. Recently this task was extended by 3D animation and audio (see "Theory versus Practice?"). Virtual artificial satellite yields following animation picture.
Following files are used for above 3D animation.
Now we would like replace earth.png texture by digital image processing result. In result we have following.
6.3.1 Installation of these samples
6.3.1.1 Installation of containes
Extract Containers.zip
directory to directory of Aviation.exe
file.
4.2.7.4 Start animation
Start Aviation.exe
.
Open OrbitImage.cfa
or OrbitTwoImage.cfa
.
- Download sample with processing of one image - 1.5 MB
- Download sample with processing of two images - 3.6 MB
Click following Animation button
Points of Interest
Writing of this article inspires recollections. When I was a young engineer I worked with very bad documents. Long time later I worked with documents of US companies. These documents were related to different areas. However the documents are clear since they had uniform style. After that I try to unify style of my own documents.