AWS SQS

· scales from 1 message per second to 10,000s per second

· Default retention of messages 4 days, maximum of 14 days

· No limit to how many messages can be in queue

· Low Latency (<10 ms on publish and receive)

· Horizontal scaling in terms of number of consumers

· Can have Duplicate messages (at least once delivery, occasionally)

· Can have out of order messages (best effort ordering)

· Limitation of 256KB per message sent

AWS SQS - Delay Queue

· Delay a message (consumers don’t see it immediately) up to 15 minutes

· Default is 0seconds (message is available right away)

· Can set a default at queue level

· Can override the default using DelaySeconds parameter

SQS - Producing Messages

· Define Body

· Add message attributes (metadata - optional)

· Provide delivery Delay (optional)

· Get back

· Message identifier

· MD5 hash of the body

SQS - Consuming Messaging

· consumers ….

· Poll SQS for messages ( receive up to 10 at a time)

· Process the message within the visibility timeout

· Delete the message using the Message ID and receipt handle

SQS - Visibility timeout

· When a consumer polls a message from the queue, the message is "invisible" to other consumers for a defined period… the Visibility Timeout:

§ Set between 0 seconds and 12 hours ( default 30 seconds)

§ If too high (15 minutes) and consumer fails to process the message, you must wait a long time before processing the message again

§ If too low (30 seconds) and the consumer needs time to process the message ( 2 minutes), another consumer will receive the message and the message will be processed more than once

· ChangeMessageVisability API to change the visability while processing a message

· DeleteMessage API to tell SQS the message was successfully processed

AWS SQS - Dead Letter Queue

· if a consumer fails to process a message within the Visibility Timeout…. the message goes back to the queue!

· We can set a threshold of how many times a message can go back to the queue - its called a "redrive policy"

· After the threshold is exceeded, the message goes into the dead letter queue (DLQ)

· We have to create a DLQ first and then designate it dead letter queue

· Make sure to process the messages in the DLQ before they expire!

AWS SQS - Short Polling

AWS SQS - Long Polling

· When a consumer requests message from the queue it can optionally "wait" for messages to arrive if there are none in the queue.

· This is called Long Polling

· LongPolling decreases the number of API calls made to SQS while increasing the efficiency and latency of the application

· The wait time can be between 1 to 20 seconds

· Long polling can be enabled at the queue level or the API level using WaitTimeSeconds

AWS SQS - FIFO Queue

· First in First Out - not available in all regions

· Name of the queue must end in .fifo

· Lower throughput ( up to 3000 per second with batching, 300/s without

· messages are processed in order by the consumer

· Messages are sent exactly once

· No per message delay (only per queue delay)

· Ability to do content based de-duplication

· 5 minute interval de-duplication using "Duplication ID"

· Message Groups

§ Possibility to group messages for FIFO ordering using "Message GroupID"

§ Only one worker can be assigned per message group so that messages are processed in order

§ Message group is just an extra tag on the message!

FIFO queues support message groups that allow multiple ordered message groups within a single queue.

Amazon SQS uses short polling by default, querying only a subset of the servers (based on a weighted random distribution) to determine whether any messages are available for inclusion in the response. Short polling works for scenarios that require higher throughput. However, you can also configure the queue to use Long polling instead, to reduce cost.

The ReceiveMessageWaitTimeSeconds is the queue attribute that determines whether you are using Short or Long polling. By default, its value is zero which means it is using Short polling. If it is set to a value greater than zero, then it is Long polling. Hence, Option 2 is correct.

Quick facts about SQS Long Polling:

-Long polling helps reduce your cost of using Amazon SQS by reducing the number of empty responses when there are no messages available to return in reply to a ReceiveMessage request sent to an Amazon SQS queue and eliminating false empty responses when messages are available in the queue but aren't included in the response.

-Long polling reduces the number of empty responses by allowing Amazon SQS to wait until a message is available in the queue before sending a response. Unless the connection times out, the response to the ReceiveMessage request contains at least one of the available messages, up to the maximum number of messages specified in the ReceiveMessage action.

-Long polling eliminates false empty responses by querying all (rather than a limited number) of the servers. Long polling returns messages as soon any message becomes available.