Skip to main content

How do we document and design our events layer?

Overview​

As we scale our event-driven architecture, it becomes increasingly difficult to keep track of where the data is routed to, which applications or clients are publishing events, and which are consuming. Because of its decoupled nature, there is no simple way to identify the correlations between these various components.

Some questions that arise from this are:

  • If I change the event payload format, what are the downstream applications affected by the change?
  • As a developer that needs a specific set of data, what does my application need to subscribe to?
  • What data is currently available to my application or client?

What is event driven architecture?​

Event-driven architecture (EDA) is an application development design used to respond to an “event” in real time. For example, think about placing an order on rapha.cc. That event determines which actions should be taken by any service after the event occurs.

For the event to trigger the correct response, an event-driven system follows three key components: event producers, event routers, and event consumers.

Event producer: A customer places an order on an "Order" event that is sent to an event router.

Event router: Makes the event available to event consumers that subscribe “Order” events.

Event consumers: Services such as "Stock" and "Email" consume the order event.

Why do we need event management?​

Spreadsheets or even architectural diagrams have been implemented to try and solve these challenges. For architectural diagrams they only give you context on the flow at that specific time and does not take into account all subscribers but rather the problem you are trying to fix for that particular squad or domain.

Adopting event management as a practise will solve the challenges of event-driven architecture by easing design, development and onboarding.

What is an Event Portal?​

An event portal is much like an API portal (think swagger) which provides a single place to design, discover, manage and visualise events.

An event management tool should be a single place for architects and developers to collaborate and document the various events routed throughout the organisation. There should be a representation of applications, events, and the payload formats for those various events.

An event portal should let you do the follow:

Define and model event-driven systems​

  • Organise your systems into application domains or clients
  • Create and/or import payload schema definitions in a variety of formats
  • Create events and their topic structure
  • Design each application’s asynchronous pub/sub interface

Visualise existing relationships​

  • Event flows crossing application domain or client
  • Application interactions
  • Event flow to and from each application

Quickly develop consistent event-driven applications​

  • Export AsyncAPI 2.0.0 specifications
  • Use code generators to broker API code for your applications

Discover and share events of interest​

  • Advertise and share events for reuse
  • Create new applications using events from other groups/teams

Govern your event-driven system​

  • Understand the lineage of event data
  • Determine impact of upcoming changes to apps, events and schemas
  • Control consumption of events across teams/application domains

Integrate with 3rd-party systems for programmatic interactions, including:​

  • Bulk importing existing EDA models
  • Integrating with CI/CD pipelines
  • Creating custom reports

Manage and audit changes to events, schemas and apps​

  • Make updates to existing apps, events, and schemas to create new revisions
  • Create new versions of apps, events and schemas by starting from the latest revision of an existing version
  • Roll back to previous object revision
  • Undo accidentally deleted objects
  • Audit and track changes to individual users

Runtime event discovery (in preview)​

  • Capture events flowing across all of your event brokers (Appliance, Software and As a Service)
  • Visualise the event topic hierarchy being used
  • Understand statistics about your events

How does an Event Portal differ from a Schema Registry?​

A schema registry helps you discover the events/messages that exist within the organization or in our case AWS environment and understand the payload schemas so you can:

  • understand if they include information your application needs, and
  • create business logic that extracts those data elements. Without this “schema” payload contract, you do not know the format of the data.

The ability to auto-discover events and let developers and architects review a menu of what’s available is useful, but to effectively deploy and manage events and event-driven applications at enterprise-scale, you also need to be able to:

  • Govern the behavior and usage of events
  • Use a wide variety of payload schemas
  • Collaborate on the design and deployment of events

An event portal should enable you to design your event-driven applications, events, and schemas in one tool, and automatically visualize them as interconnected network diagrams that we can go over in design reviews.

Risks​

The biggest issue we have is that we already have a large amount of clients and events deployed in an event-driven manner. We don’t have time to manually document all of these objects. We need to find a way to automate the discovery of events and clients. This would allow us to populate a catalog and visualise relationships without manual effort.

Solutions​

Below is an initial list of solutions that could help tackle the problem. Please read and research on all of them so we can make an infomed decision on which solution is best for Rapha.

EventCatalog​

EventCatalog is an Open Source project that helps you document your events, upstream/downstream services and domains. Domains are the same as our term clients which is just a way to group events and services by application domain.

You can also generate documentation from any third party system using plugins. Although EventCatalog doesn't have any design capability you are able to visualise your EDA. You are also able to render code examples, node graphs, event schemas, event versions, event owners and much more.

AsyncAPI​

AsyncAPI is a set of open source tools to help teams easily build and maintain an event driven architecture. It is powered by the AsyncAPI specification which is the the industry standard for defining asynchronous APIs. The current feature set includes

Specifictions​

Allows you to define the interfaces of asynchronous APIs and is protocol agnostic.

Document APIs​

A set of tools to generate documentation at the build level, on a server, and on a client.

Code Generation​

Generate documentation, Code (TypeScript, Java, etc), and more out of your AsyncAPI files.

EventBridge Atlas​

EventBridge Atlas essentially generates documentation from schemas. It does this by using the aws-sdk and discovering schemas in a particular registry. It then uses a parser to generate static HTML. Its a great tool but only gives you a particular view into your EDA and is no a central place to perform all the actions needed such as designing and visualising.

Solace​

Solace offers a range of products to help with the management of your EDA.

Platform​

PubSub+ Platform is an event streaming, management and monitoring platform that gives you everything you need to design, deploy and manage an event-driven system.

  • Stream events across hybrid, multi-cloud and IoT environments, quickly, reliably and securely
  • Manage your entire events ecosystem
  • Design and develop event-driven apps and microservices faster

Cloud​

PubSub+ Platform available as a service in your favourite public and virtual public clouds and on premises Kubernetes environments.

Event Broker​

Stream events across your enterprise quickly, reliably, and securely.

PubSub+ lets you connect event brokers to form an event mesh—an architecture layer that allows you to dynamically route events from one application to any other application no matter where those applications are deployed (no cloud, private cloud, public cloud)—so you can connect and orchestrate microservices, push events from on-premises systems of record to cloud services, and enable digital transformation across LoBs and IoT.

Event Portal​

Design, discover, visualize, catalog and govern your entire events landscape.

Resources​