Skip to main content

Spike - Replace manifest files to automate digital assets sync with Hybris

Overview​

Results from a spike looking at the SDKs and APIs provided by Cloudinary that we could use to replace the current process by which we sync our DAM (Amplience) with Hybris. The aim of the spike was to review the APIs and SDKs and come up with some possible solutions that we can move forward with for a working POC.

Node.js SDK​

We will be using the Node.js SDK which provides image and video upload, transformation, optimisation and delivery capabilities. The SDKs wrap Cloudinary's REST APIs and offer a variety of useful helper methods.

Search API​

The Search API allows us to fetch information on all the assets in our account, with granular filtering options applied via query expressions. For example we could use this API to fetch all assets that have been updated since a certain period, we can also filter by file type, size and many other dimensions. This is part of the Admin API which is a rate limited API, with rates depending on your plan - the default on the free plan is 500 requests per hour which should suit our current needs.

Notifications​

Cloudinary provides a notification feature that sends a HTTP POST request to a notification URL that we provide, when certain actions have been completed e.g. image upload. The data will contain all results related to that action. We can configure a Global notification URL in our Cloudinary dashboard, and once we've done that Cloudinary will send responses to that address any time a user performs an update in the media library, or any time we run any Cloudinary Upload API method.

Solution 1 - Synchronously fetch resources from Cloudinary via the Search API​

  1. Cloudwatch alarm will trigger Lambda to fetch assets from Cloudinary Search API at a specified time interval - hourly, daily etc
  2. Cloudinary Search API will return assets
  3. Response is transformed in Lambda function
  4. Data is sent to Hybris from Lambda

Notes:

  • Not recommended, sync and multiple points of failure

Cloudinary - Hybris asset sync design - sync

Solution 2 - Asynchronously fetch resources from Cloudinary via the Search API​

  1. EventBridge rule will trigger Step Function every 5 mins, which targets a Lambda that will then fetch assets from the Cloudinary Search API and wait for it to resolve. The Step Function will automatically retry if the request fails or times out.
  2. The returned response is transformed by another Lambda in the Step Function
  3. Once the response has been transformed, the data is passed into the Platform Image Notifcation SQS and then onto EventBridge. The SQS which has a mechanism to prevent us losing any data at this stage.
  4. Data is passed to EventBridge, which Hybris is a consumer of. Between EB and Hybris there is the Hybris Image Ingest SQS with the same mechanism to prevent any loss of data if Hybris is unavailable or any requests fail.
  5. Data is sent to Hybris

Notes:

  • Any other potential consumers in future can also listen to EB in future
  • We need to fine tune the time interval to fetch data, and bear in mind rate limits
  • Step Function easy to monitor and more flexible than just a Lambda by itself
  • Response data is consistent with every request, means that we don't need to manipulate it or cover multiple cases

Cloudinary - Hybris asset sync design - async with stepfunction

Solution 3 - Asynchronously subscribe and receive resource updates from Cloudinary via Notifications and webhooks​

  1. Cloudinary notification is sent to API Gateway via webhooks URL
  2. The returned response is passed into the Platform Image Notifcation SQS and then onto EventBridge as above.
  3. Following steps are the same as above (EB -> Hybris Image Ingest SQS -> Hybris)

Notes:

  • Can Hybris consume real time data?
  • Notifications are sent for all event types, we would need to filter the events and direct them to the relevant place ourselves
  • Notifications have a slightly different data structure depending on the event that has occurred, so we need to normalise the data in the Cloudinary image process Lambda before passing it on to Hybris. We need to decide which events we are going to "catch" data for to pass to Hybris, review the data structures and agree on what fields are the most important and which must be passed on to Hybris in the agreed format.
  • We need to agree on a folder structure in Cloudinary for the web team to use, similarly to what we are using for Amplience currently, that will allow us to easily get identifiable information about the assets from the asset path, such as product SKU, if it is a Gallery or Mannequin image etc.
  • This is the preferred solution, if it is possible with Hybris

Cloudinary - Hybris asset sync design - async

Resources​

https://cloudinary.com/documentation/node_integration

https://cloudinary.com/documentation/search_api

https://cloudinary.com/documentation/notifications#notification_response_examples