In the previous post we talked about Event Sourcing, its definition, components, benefits, techniques, etc… so you can get your development journey of an ES based system started. Next sensible step, in order to get into the ES mindset, is to go through the thinking process of building a real system (as real as possible) with ES pattern.
In this post I am demonstrating a fully fleshed Lap Tracking System in Motorsports built with ES pattern. The system will be built as a standalone web application. Everything will take place in memory no IO bound operations and databases are involved and thats for the sake of simplicity of deployment. The design challenges and concepts are still applicable though. It will utilise Querying, Projection, and Snapshotting techniques as discussed in previous post.
In motorsports a lap time denotes the time taken to travel from point 1 to point 2. Since the car goes into circles around a track, point 1 & 2 are the same.
A track has 3 locations where once the car goes past them an event is recorded. Theses events are called lap triggers. A lap is a logical representation of a list of events. Hence, Event Sourcing as technical solution is is a natural fit.
- The car starts and ends in its pit lane garage. This marks the start and end of a RUN.
- At least 2 consecutive events are required to define a lap (start and end times)
- Lap Duration = Current Trigger Timestamp – Previous Trigger Timestamp
- The lap type is defined based on the lap trigger id/type as summarised in the table below:
|First Trigger||Second Trigger||Lap Type|
*Installation lap is not used in this demo.
The event sourcing will have to deal with the following events.
NOTE: Don’t get confused between events and messages.
- Events are persisted to event store and are the source of truth.
- Messages are used to communicate events or other actions to the business logic. Messages encapsulate the event as a payload and can be published either by event store or a messaging system (e.g. subscriber/handler logic).
Lets have a look at what we are building utilising ES. The presentation layer should provide users with the following information for live telemetry:
- Laps per run
- All runs
- Accumulated aggregates (min, max, mean) across all laps.
- e.g. Min value describes Fastest lap time, and Max the slowest
Next I will explain about design choices and considerations for each requirement starting from the simplest one. The main workflow of incoming events all the way to consumer is depicted below.
Table below shows a dummy EventStore with the last 2 runs worth of events. The purpose of the table is to help you picture things better. Imagine this is a timeseries database where the key is the ordered timestamp. This improves look up times.
1. Laps per Run
- Querying: Every time the subscriber receives a LapTriggerred event it queries the event store for all events since last RunStarted event.
- Projection: It looks for 2 valid consecutive
LapTriggerred events and it works out its type and duration based on the table under Business Rules section.
- View Materialisation: Lap objects are appended to an in-memory table for consumption by UI.
- It is recommended to keep track of the last
RunStarted to speed up querying in order to reduce amount of events to traverse.
- Some level of validation takes place in laps builder code. We need to take into account scenarios where events are out of order because of backfills or out of order LapTriggerred events; 2 consequtive
LapTriggerred events of type the same trigger.
2. All Runs
- Querying: Every time the subscriber receives a RunEnded event, it queries the event store for all RunStarted & RunEnded events.
- Projection: Every pair of RunStarted & RunEnded events define a run. Subtracting their timestamps gives us duration.
- View Materialisation: Add new generated run to its read model storage.
- The run id is a field generated by the system not coming from the event originator (track telemetry infrastructure). The run id gets cached so other projections related to the same run can reuse it.
- Wouldn’t it be more efficient to use a snapshot instead of every time querying all events? Yes, and is demonstrated in next requirement.
3. Accumulated Aggregates
- Message Publishing: Every time a lap is projected (requires at least to lap events), as per requirement #1, the subscriber will publish a message denoting lap completion e.g. LAP_COMPLETED, and a new message handler, responsible for calculating lap aggregations, will react to it.
- Every time a RunEnded event is received by it handler, it checks for certain criteria if they are met (every 2 runs in this demo), at a new message e.g. TAKE_SNAPSHOT is published.
- Querying: The aggregate function is calculated across all laps. So upon receiving the lap completion message, we need to either query all lap events from events store and rebuild events, the already projected laps from their materialised view, or lap event from the store from a recent point in time. In my case I went for the latter by benefiting from the snapshot technique.
- Since aggregation is meant to work on accumulated results, we have to query the last accumulated result which was stored at the time of last snapshot was taken.
- Projection: Apply aggregation logic on a small subset of lap events which are since the last snapshot taken.
- Snapshot: Every time the take snapshot message is received we cache the aggregate results and their timestamp.
- View Materialisation:
- The final aggregate results are stored into the materialised view databases (aka read models)
- Save the snapshot details again into the read models storage (remember in the above app everything is kept in memory). The snapshot details are latest aggregate results & timestamp.
- Taking a snapshot is a technique to speed up look ups, and overall performance. The frequency of taking snapshots depends on the context.
- Because snapshots are used to speed up querying and calculation, it key to keep results somewhere easily accessible e.g. in memory, or local db, etc…
In the previous post, Just Event Sourcing under What’s Next section, I mentioned about considering the more complex use cases like versioning, or repeated message. Although I don’t cover that in this post, Hopefully I will dedicate another post for it, but in production we have to take them into account and be prepared, because sooner or later we will be hit by these edge cases.
If you noticed under the considerations section of each requirement, there is pretty much a desicion to be made on whether to fetch the raw data from event store or from materialised views. Its a classiscal trade off between speed and consistency. Most ES systems tend to follow eventual consistency, as there is always a delay between event saving, publishing, processing, and materialising.