SQS and Lambda

• SQS and Lambda
  • SQS
  • Lambda
  • SQS & Lambda
  • Scaling
    • Lambda Parameters
  • SQS Parameters

SQS

Simple Queue Service (SQS) is a lightweight, fully managed message queueing service. It enables you to decouple and scale serverless applications, distributed systems and microservices. With SQS, it is easy to send, receive, store and process messages between services without losing data.

Lambda

Lambdas are a fully managed, auto scaling service that lets you run code in a variety of different languages in a serverless manner. You can orchestrate the invocations or configure your function to execute on response to events.

SQS & Lambda

When you configure a SQS as an event source for a Lambda, your Lambda functions are automatically triggered when messages arrive to the SQS queue. This enables your Lambda service to automatically scale up and down based on the number of messages in the queue. The polling, reading and removing of messages from the queue will be automatically handled automatically. Successfully processed messages are removed from the queue whilst any that fail are returned to the queue or forward to a DQL depending on how many retries you have configured.

Scaling

Lets first go over some of the params that have an impact on how SQS + Lambdas scale together:

Lambda Parameters

Reserved concurrency limit: The number of executions of the Lambda function that can run simultaneously. There is an account specific limit how many executions of Lambda functions can run simultaneously (by default 1,000) and it is shared between all your Lambda functions. By reserving part of it for one function, other functions running at the same time can’t prevent your function from scaling.

BatchSize: The maximum number of messages that Lambda retrieves from the SQS queue in a single batch. Batchsize is related to the Lambda event source mapping, which defines what triggers the Lambda functions. In this case they are triggered from SQS.

SQS Parameters

ReceiveMessageWaitTimeSeconds: The length of time, in seconds, for which a ReceiveMessage action waits for a message to arrive. Valid values: An integer from 0 to 20 (seconds). Default: 0. In other words this is the time the poller waits for a new message. The default value is 0 which means it is constantly polling messages.

VisibilityTimeout: The visibility timeout for the queue, in seconds. Valid values: An integer from 0 to 43,200 (12 hours). When a poller reads a message from the SQS queue, that message still stays in the queue but becomes invisible for the period of VisibilityTimeout. During this time the read message will be unavailable for any other poller trying to read the same message and gives the initial component time to process and delete the message from the queue.

maxReceiveCount: The number of times a message is delivered to the source queue before being moved to the dead-letter queue. When the ReceiveCount for a message exceeds the maxReceiveCount for a queue, Amazon SQS moves the message to the dead-letter-queue.

Lambda uses long polling to poll messages in the queue, which means that it does not poll the queue all the time but instead on an interval between 1 to 20 seconds, depending on what you have configured to be your queue’s ReceiveMessageWaitTimeSeconds. Lambda service’s internal poller reads messages as batches from the queue and invokes the Lambda function synchronously with an event that contains a batch of messages. The number of messages in a batch is defined by the BatchSize that is configured in the Lambda event source mapping.

When messages start arriving to the queue, the Lambda reads (up to) the first 5 batches. It will then invokes a function per batch. If there are still messages available, the number of processes reading the batches are increased by up to 60 more instances per minute.

By setting up a reserved concurrency limit for your Lambda, you guarantee that it will get part of the account’s Lambda resources at any time, but at the same time you also limit your function from scaling over that limit, even though there would be Lambda resources available for your account to use. When that limit is reached and there are still messages available in the queue, one might assume that those messages will stay visible in the queue, waiting until there’s free Lambda capacity available to handle those messages. Instead, the internal poller still tries to invoke new Lambda functions for all the new messages and therefore causes throttling of the Lambda invokes.

Throttling of valid messages can be avoided with the following considerations:

• Be careful when configuring a reserved concurrency to your Lambda function: the smaller the concurrency, the greater the chance that the messages get throttled. AWS suggests the reserved concurrency for a Lambda function to be 5 or greater.

• Set the maxReceiveCount big enough in your SQS queue’s properties, so that the throttled messages will eventually get processed after the burst of messages. AWS suggest you set it to 5 or greater.

• By increasing message VisibilityTimeout of the source queue, you can give more time for your Lambda to retry the messages in the case of message bursts. AWS suggests this to be set to at least 6 times the timeout you configure to your Lambda function.