Integrating using the new API Destinations of Amazon EventBridge

Amazon EventBridge is the central event bus in your AWS-based environment. A comprehensive integration messaging service that includes error handling, filtering, content-based routing, replay functionality, event schemas, and OTB SaaS integration capabilities. This in the form of a developer-friendly integration service ingested with iPaaS capabilities.

March 2021 AWS extended the service with the “API Destinations” feature. This allows integrating incoming events towards downstream REST APIs without the need to code a Lambda function. Further reducing complexity, development time and costs. In this blog, I’ll try out the API Destination functionality using 3 commonly used downstream services: Salesforce, Mailchimp, and MS Teams.

Integration architecture with Mailchimp, Salesforce, and MS Teams as targets and simplified ingestion flow from Magento.

From the depicted diagram, you may already conclude that not all 3 connections were successful. More on that later …

Downstream service ingestion rate control
Interestingly, “API Destinations” allows you to control the ingestion rate into the downstream services. This is very crucial for enterprise integration because not all downstream services scale like a Lambda or DynamoDB. This feature provides a significant simplification of the common “Eventbridge -> SQS -> Lambda -> Downstream service” pattern. A Pattern where SQS is used in combination with Lambda to control the ingestion rate. And Lambda by itself for Transformation, Enrichment, and connectivity.

Today Amazon EventBridge takes care of rate control, transformation, and REST API connectivity. At least for some scenario

Let’s build the proposed integration solution
Let’s validate if the service lives up to its promise. If you plan to follow along, the tutorial prerequisites:
• Magento “New Customer” events published into a custom event bus
• Mailchimp account with API Key
• MSTeams channel webhook URL

Note that the web console allows us to create the full setup. Though in this blog we’ll use CloudFormation. The used template: https://gist.github.com/Samuelvdc/96d1f0c91eee6b7fed58a3dc755f5704#file-cloudformation-eb-apidestinations-demo-yaml

Creating the “AWS::Events::Connection” resources

There are currently 3 supported authorization types: Basic authentication, API Key and OAuth Client credentials flow. It’s also possible to set static (or from the event payload) HTTP headers, query parameters or body fields.

For both Mailchimp (Basic Auth) and MS Teams (no Auth needed) the setup is straightforward. Unfortunately, Salesforce does not support any of the currently supported authorization types. To push to Salesforce, we still need an intermediate Lambda.

Creating the “AWS::Events::ApiDestination” resources

Using the API Destination resource you’ll configure the actual REST API endpoint of the downstream service. You should only provide the method and endpoint for this. Also, the discussed rate-limiting capability is configured here.

Two thoughts on this:
There is no support for dynamically creating the “InvocationEndpoint”. For example, I wanted to upsert a member in Mailchimp. So it also pushes changes of the customer to Mailchimp. For this you need to use following endpoint: PUT /lists/{list_id}/members/{subscriber_hash}.
• The “subscriber_hash” id is the MD5 hash of the email address. This requires custom code and still requires a Lambda function.
• Is it possible to map “list_id” from the event payload? I haven't found a possibility yet (though for this scenario not relevant).

Secondly, it is not possible to define what a successful call means. EventBridge will assume this from the returned HTTP status code. In most cases this will not be an issue, but it’s certainly something to be aware of.

Creating the “AWS::Events::Rule” resource

Nothing new here, I’ve used the Input Transform functionality to build a new JSON payload with input from the incoming event. Note that this demo CloudFormation template doesn’t include a dead letter queue integration for messages which failed to deliver. When using this in production, look into that.

Afterthoughts
API Destinations is an interesting addition to the EventBridge offering. And paves the way to further simplify Event-driven integration architectures.

The current capabilities are rather limited for most scenarios to eliminate the need for Lambda functions: basic transformations, only JSON based, no hooks for integration business monitoring, no response inspection, no enrichment. However, this addition strengthens the positioning of the service for enterprise integration use cases. And for some simple scenarios, it already provides direct added value.

The rate control is a required capability when integrating with downstream services. It’s very interesting to see this queuing capability incorporated in the EventBridge offering. As of last week, the service was still a Push-Push only messaging service. Hopefully, the rate control capability will also be introduced for a Lambda target. Enabling an “Eventbridge -> Lambda -> non-scalable downstream service” pattern.

Below would be the current solution architecture for the initially proposed use case:

Integration architecture without DLQs.