|
..with the code to allow you to specify the backing storage to use fro any given aggregate class in the application config - so much refactoring is needed.
Maybe I'd better take a break from it for now...
|
|
|
|
|
I'm doing a talk for the Dublin MicroServices user group[^] using my CQRS on Azure project to demonstrate the concepts of event sourcing (and also CQRS) on Oct 28th.
(Hopefully the software is nearer ready by then)
modified 14-Jun-17 14:37pm.
|
|
|
|
|
The MacBook air is going to the farm upstate (where it will have a happy life but we won't be able to visit) and I'm thinking I might replace it with an ASUS ZenBook UX303 variant. Still playing with the spec list though - like a child in a toy shop.
|
|
|
|
|
I got an HP Envy in the end with B&O speakers - currently testing it with a bit of Fields[^]..
|
|
|
|
|
One of the design goals I am looking at with regard to the CQRS on Azure project is that it should allow the solution to any performance issue to be "just throw more hardware at it" (or, as this is to be hosted on Azure the more realistic solution would be "just spin up more virtual machines").
This means that you have to take considerable care that this adding of new hardware does not cause any performance issues of its own - therefore it must not create nor further tax any bottlenecks.
To achieve this I have based the architecture for handling the event streams (and running projections over them as well as maintaining the "identity groups") on a peer-to-peer mesh and used asynchronous message passing to communicate between hosts.
Hosts may pass any request they receive onwards to another host if they are under too high a load. The request message has a reference to the originator so that when a host is found that can respond, that response can be sent straight back without having to unwind the path it took to get there.
This does, of course, mean that requests must have an unique identifier and since I don't want any bottleneck in the production of this uniqueness I use a GUID for each message as it is created.
There are also notifications that hosts can send each other - such as "new host joining", "host shutting down" etc. and also an acknowledgement message can be sent by a host that receives a request to allow the sending host to know that it has been received. If a sending host does not get an acknowledgement response in a given time frame then it can send the request to a different target to try that instead.
Actual data updating commands are a trickier proposition as you either have to ensure that the command is idempotent so if it is applied more than once it doesn't cause a problem or you have to have some sort of way of knowing if a command has been executed or not. (This latter solution would introduce a bottleneck.)
|
|
|
|
|
In order to get the most out of the highly distributed and highly parallelized nature of the Azure back end used for the Event Stream (in particular when using either append blobs or file based storage) it makes sense to have a number of instances performing operations on those event streams simultaneously.
In my project I refer to these as "Hosts" but process or server is equally valid.
These hosts are arranged in a peer-to-peer mesh where no one host is either a bottleneck nor a single point of failure risk. There is also the requirement for a host to be allowed to pass a request onwards if it is too busy (or not best suited) to service it.
Public Enum RequestCategories
NotSet = 0
ExecuteCommand = 1
ExecuteQuery = 2
GetIdentityGroupMembers = 3
RunProjection = 4
RunClassifier = 5
End Enum
The response messages can indicate acknowledgement, or an error state return or can return the data requested. ( In the case of the command there is no data requested but a "command complete" return is needed).
There are also notifications needed for the maintenance of the peer-to-peer mesh network: host starting, host stopping, host high throughput notification and so on.
Any feedback before I bake this in to the CQRS on Azure project?
|
|
|
|
|
I have made the IProjection interface support the concept of caching the state of the projection as at any given point in time so that the running of a projection over a very large event stream can be speeded up - especially in situations where multiple queries are essentially asking the same question.
Public Overridable Sub LoadFromSnapshot(snapshotToLoad As IProjectionSnapshot(Of TAggregate, TAggregateKey))
Implements IProjection(Of TAggregate, TAggregateKey).LoadFromSnapshot
If (snapshotToLoad IsNot Nothing) Then
m_CurrentSequenceNumber = snapshotToLoad.Sequence
If (snapshotToLoad.AsOfDate > m_CurrentAsOfDate) Then
m_CurrentAsOfDate = snapshotToLoad.AsOfDate
End If
m_currentValues.Clear()
m_currentValues.AddRange(snapshotToLoad.Values)
End If
End Sub
This can be done by storing the projection values as name:value pairs or by explicitly hard wiring the snapshot values to the projection properties - I prefer the former but make no exclusion.
|
|
|
|
|
LegacySitePackage failed for package [Microsoft.VisualStudio.Editor.Implementation.EditorPackage]
Source: 'mscorlib' Description: An item with the same key has already been added. System.ArgumentException: An item with the same key has already been added. at System.ThrowHelper.ThrowArgumentException(ExceptionResource resource) at System.Collections.Generic.Dictionary`2.Insert(TKey key, TValue value, Boolean add) at Microsoft.VisualStudio.ExtensibilityHosting.VsExportProviderFactory.<>c__DisplayClass54_0.<CreateAssemblyCatalogsAsync>b__6(KeyValuePair`2 namedCatalog) at System.Threading.Tasks.Dataflow.ActionBlock`1.ProcessMessage(Action`1 action, KeyValuePair`2 messageWithId) at System.Threading.Tasks.Dataflow.ActionBlock`1.<>c__DisplayClass5.<.ctor>b__0(KeyValuePair`2 messageWithId) at System.Threading.Tasks.Dataflow.Internal.TargetCore`1.ProcessMessagesLoopCore()
--- End of stack trace from previous location where exception was thrown
--- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.ExtensibilityHosting.VsExportProviderFactory.<CreateAssemblyCatalogsAsync>d__54.MoveNext()
--- End of stack trace from previous location where exception was thrown
--- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter.ValidateEnd(Task task) at Microsoft.VisualStudio.ExtensibilityHosting.VsExportProviderFactory.<GetCurrentAssemblyCatalogsAsync>d__31.MoveNext()
--- End of stack trace from previous location where exception was thrown
--- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.ExtensibilityHosting.VsExportProviderFactory.<GetExportProviderFactoryAsync>d__23.MoveNext()
--- End of stack trace from previous location where exception was thrown
--- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.ComponentModelHost.ComponentModel.<GetMEFV3ExportProviderInternalAsync>d__49.MoveNext()
--- End of stack trace from previous location where exception was thrown
--- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at System.Runtime.CompilerServices.TaskAwaiter`1.GetResult() at Microsoft.VisualStudio.ComponentModelHost.ComponentModel.<GetMEFV3ExportProviderWrapperAsync>d__48.MoveNext()
--- End of stack trace from previous location where exception was thrown
--- at System.Runtime.CompilerServices.TaskAwaiter.ThrowForNonSuccess(Task task) at System.Runtime.CompilerServices.TaskAwaiter.HandleNonSuccessAndDebuggerNotification(Task task) at Microsoft.VisualStudio.Threading.JoinableTask.CompleteOnCurrentThread() at Microsoft.VisualStudio.Threading.JoinableTask`1.CompleteOnCurrentThread() at Microsoft.VisualStudio.Threading.JoinableTaskFactory.Run[T](Func`1 asyncMethod, JoinableTaskCreationOptions creationOptions) at Microsoft.VisualStudio.Threading.JoinableTaskFactory.Run[T](Func`1 asyncMethod) at Microsoft.VisualStudio.ComponentModelHost.ComponentModel.GetLazyValue[T](AsyncLazy`1 lazy) at Microsoft.VisualStudio.ComponentModelHost.ComponentModel.GetService[T]() at Microsoft.VisualStudio.Editor.Implementation.EditorParts.get_ContentTypeRegistryService() at Microsoft.VisualStudio.Editor.Implementation.LanguageServiceToContentTypeMapper.MakeLanguageServiceContentTypes(SettingsStore settingsStore) at Microsoft.VisualStudio.Editor.Implementation.LanguageServiceToContentTypeMapper.InitLanguageServiceToContentTypeMapper(IServiceProvider serviceProvider) at Microsoft.VisualStudio.Editor.Implementation.EditorPackage.Initialize() at Microsoft.VisualStudio.Shell.Package.Microsoft.VisualStudio.Shell.Interop.IVsPackage.SetSite(IServiceProvider sp)
|
|
|
|
|
I think every project has to go through that phase where I'm listening to "The Reptile House E.P." while programming - CQRS on Azure just reached that point.
|
|
|
|
|
Every morning that I wake out of Africa I am in exile.
|
|
|
|
|
I'm doing a very short splash-and-dash demo of the CQRS designer project at a Dublin meet up[^] on Thursday.
Looking forward to some feedback.
|
|
|
|
|
..and finally got the code generating attributes to classify the events..
<CQRSAzure.EventSourcing.DomainNameAttribute(domainNameIn:="TennisTournament"), _
CQRSAzure.EventSourcing.Category(categoryNameIn:="Match Status")> _
Partial Public Class GameFinished
Quote: )»Ŧ»)
modified 8-Apr-16 8:32am.
|
|
|
|
|
Started trying to split the TFS project and started wake[^] at the same time.
Got to the fix at 47 minutes - spooky, eh?
|
|
|
|
|
Definitely the aggregate needs to control writing to the event stream but maybe there are scenarios where the event stream can be read without the aggregate?
(Hmm - philosophers hat needed)
|
|
|
|
|
In the end it seems best if it does - and also a public method for each event that the aggregate can handles so that it is easily mocked up..
Option Strict Off
Option Explicit On
Imports CQRSAzure
Imports CQRSAzure.Aggregation
Imports CQRSAzure.EventSourcing
Namespace Football_League.Game
Partial Public Class Game
Inherits Object
Implements IGame
Private _Key As System.Guid
Private m_eventStream As IEventStream(Of IGame)
Sub New()
MyBase.New
End Sub
Sub New(ByVal Key_In As Object)
MyBase.New
_Key = Key_In
End Sub
Public Function GetAggregateIdentifier() As String Implements IAggregationIdentifier.GetAggregateIdentifier
Return _Key.ToString
End Function
Public Sub SetKey(ByVal Key_In As System.Guid) Implements CQRSAzure.EventSourcing.IAggregationIdentifier(Of System.Guid).SetKey
_Key = Key_In
End Sub
Public Sub AddEvent(ByVal eventToAdd As IEvent(Of IGame)) Implements IEventStream(Of IGame).AddEvent
m_eventStream.Add(eventToAdd)
End Sub
Public Sub Scheduled(ByVal eventToAdd As IScheduled)
End Sub
Public Sub Rescheduled(ByVal eventToAdd As IRescheduled)
End Sub
Public Sub Started(ByVal eventToAdd As IStarted)
End Sub
End Class
End Namespace
|
|
|
|
|
Identity groups need some way of evaluating if any given identity is in or out of the group based on the event stream of that entity.
This is somewhat similar to how a projection works but I think to avoid confusion I shall call this specific case of a projection a "classifier".
(I'm probably going to rely on the magic of inheritance to make the code underlying this and the code underlying a projection match)
|
|
|
|
|
Classifiers
A classifier is a special form of a projection which runs over the event stream of an aggregate and decides if it is "in" or "out" of the identity group as at any given point in time.
In the above example of the "accounts held by US citizens" identity group the classifier might run over events occurring to the aggregate identifier "account" and decide if the account is in or out of the group based on "account opened", "beneficial ownership changed" and any other similar events.
|
|
|
|
|
Have the very early cut of the code generation from the CQRS DSL designer working now...
'------------------------------------------------------------------------------
' <auto-generated>
' This code was generated by a tool.
' Runtime Version:4.0.30319.42000
'
' Changes to this file may cause incorrect behavior and will be lost if
' the code is regenerated.
' </auto-generated>
'------------------------------------------------------------------------------
Option Strict Off
Option Explicit On
Imports CQRSAzure
Imports CQRSAzure.Aggregation
Imports CQRSAzure.EventSourcing
Namespace Football_League.Player.eventDefinition
'''<summary>
'''Player registered with the league
'''</summary>
'''<remarks>
'''This will generate the player's unique regsitration id
'''</remarks>
Partial Public Class Registered
Inherits Object
Implements IRegistered
#Region "Private members"
Private _PreviousClub As Date
Private _Amateur As Boolean
Private _DateOfBirth As Date
#End Region
'''<summary>
'''Empty constructor for serialisation
'''This should be removed if serialisation is not needed
'''</summary>
Sub New()
MyBase.New
End Sub
'''<summary>
'''Create and populate a new instance of this class from the underlying interface
'''</summary>
'''<remarks>
'''This should be called when the event is created from an event stream
'''</remarks>
Sub New(ByVal RegisteredInit As IRegistered)
MyBase.New
_PreviousClub = RegisteredInit.PreviousClub
_Amateur = RegisteredInit.Amateur
_DateOfBirth = RegisteredInit.DateOfBirth
End Sub
'''<summary>
'''Create and populate a new instance of this class from the underlying properties
'''</summary>
'''<remarks>
'''This should be called when the event is created from an event stream
'''</remarks>
Sub New(ByVal PreviousClub_In As Date, ByVal Amateur_In As Boolean, ByVal DateOfBirth_In As Date)
MyBase.New
_PreviousClub = PreviousClub_In
_Amateur = Amateur_In
_DateOfBirth = DateOfBirth_In
End Sub
'''<summary>
'''Where did the player come from
'''</summary>
Public ReadOnly Property PreviousClub() As Date
Get
Return _PreviousClub
End Get
End Property
'''<summary>
'''Is the player registered as an amateur
'''</summary>
Public ReadOnly Property Amateur() As Boolean
Get
Return _Amateur
End Get
End Property
'''<summary>
'''Player's date of birth
'''</summary>
Public ReadOnly Property DateOfBirth() As Date
Get
Return _DateOfBirth
End Get
End Property
End Class
End Namespace
There's a long way to go from here - but this is definitely heading in the right direction!
|
|
|
|
|
although - code generation does result in some ugly variable names
'''<summary>
'''Create and populate a new instance of this class from the underlying properties
'''</summary>
'''<remarks>
'''This should be called when the event is created from an event stream
'''</remarks>
Sub New(ByVal Start_In As Date, ByVal Court_In As Integer, ByVal Umpire_In As String, ByVal Player_s__1_In As String, ByVal Player_s__2_In As String)
MyBase.New
_Start = Start_In
_Court = Court_In
_Umpire = Umpire_In
_Player_s__1 = Player_s__1_In
_Player_s__2 = Player_s__2_In
End Sub
|
|
|
|
|
More code generation - this time for factory methods...
Shared Function Create(ByVal Date_Of_Birth_In As Date,
ByVal Observations_In As String,
ByVal Heiffer_Identifier_In As String) As IBorn
Return New Born(Date_Of_Birth_In, Observations_In, Heiffer_Identifier_In)
End Function
|
|
|
|
|
And code generation for projections....
Option Strict Off
Option Explicit On
Imports CQRSAzure
Imports CQRSAzure.Aggregation
Imports CQRSAzure.EventSourcing
Imports Herd.Cow
Imports Herd.Cow.eventDefinition
Namespace Herd.Cow.projection
Partial Public Class Location
Inherits Object
Implements ILocation
#Region "Private members"
Private _In_Shed As Boolean
Private _Location As String
#End Region
Public ReadOnly Property In_Shed() As Boolean Implements ILocation.In_Shed
Get
Return _In_Shed
End Get
End Property
Public ReadOnly Property Location() As String Implements ILocation.Location
Get
Return _Location
End Get
End Property
Public Overloads Sub HandleEvent(ByVal eventToHandle As IMoved_To_Field) Implements CQRSAzure.EventSourcing.IHandleEvent(Of IMoved_To_Field).HandleEvent
_In_Shed = False
_Location = eventToHandle.Moved_To
End Sub
Public Overloads Sub HandleEvent(ByVal eventToHandle As IMoved_To_Shed) Implements CQRSAzure.EventSourcing.IHandleEvent(Of IMoved_To_Shed).HandleEvent
_In_Shed = True
_Location = eventToHandle.Shed_Name
End Sub
End Class
End Namespace
|
|
|
|
|
For a query definition, two classes are created. The "Definition" class is in charge of specifying the model for the input parameters and return parameters (as would be used in the MVVM / MVC front end):-
Imports CQRSAzure
Imports CQRSAzure.Aggregation
Imports CQRSAzure.EventSourcing
Imports CQRSAzure.QueryDefinition
Imports Herd.Cow
Namespace Herd.Cow.queryDefinition
Partial Public Class Get_Herd_Location_Definition
Inherits QueryDefinitionBase(Of IEnumerable(Of IGet_Herd_Location_Definition_Return))
Implements IGet_Herd_Location_Definition
Private Property As_Of_Date() As Date Implements IGet_Herd_Location_Definition.As_Of_Date
Get
Return MyBase.GetParameterValue("As Of Date", 0)
End Get
Set
MyBase.SetParameterValue("As Of Date", 0, Value)
End Set
End Property
End Class
End Namespace
And the "Handler" does the actual work of populating the return data for a query:-
Option Strict Off
Option Explicit On
Imports CQRSAzure
Imports CQRSAzure.Aggregation
Imports CQRSAzure.EventSourcing
Imports CQRSAzure.QueryDefinition
Imports CQRSAzure.QueryHandler
Imports Herd.Cow
Namespace Herd.Cow.queryHandler
Partial Public Class Get_Herd_Location_Handler
Inherits Object
Implements IGet_Herd_Location_Handler
Public Function HandleQuery() As IEnumerable(Of IGet_Herd_Location_Definition_Return) Implements IGet_Herd_Location_Handler.HandleQuery
Dim queryReturn As IEnumerable(Of IGet_Herd_Location_Definition_Return) = Nothing
Return queryReturn
End Function
End Class
End Namespace
|
|
|
|
|
It turns out that code generation for "Classifiers" is the hardest part.. so far I have:-
Imports CQRSAzure.EventSourcing
Imports CQRSAzure.IdentifierGroup
Imports Football_League.Team
Namespace Football_League.Team.classifier
Partial Public Class Is_Premier_League
Inherits Object
Implements IIs_Premier_League
Sub New()
MyBase.New
End Sub
Private Overloads Function Evaluate(ByVal eventToEvaluate As IRelegated) As EvaluationResult
Implements CQRSAzure.IdentifierGroup.IClassifierEventHandler(Of IRelegated).Evaluate
Return
End Function
Private Overloads Function Evaluate(ByVal eventToEvaluate As IPromoted) As EvaluationResult
Implements CQRSAzure.IdentifierGroup.IClassifierEventHandler(Of IPromoted).Evaluate
End Function
End Class
End Namespace
|
|
|
|
|
...harder than it should be
|
|
|
|
|
So the plan is for a visual tool with which you create your CQRS model and have it CodeGen the required classes for events, projections, queries and commands - and the required documentation (both as code comments and as HTML files for the business users)
Progress is slow but steady - the diagramming is basically there but with some rough edges (using the modelling SDK) and I have started on the code generation using CodeDOM.
I reckon I'm about a month away from a serviceable beta.
(All help gratefully received)
|
|
|
|