php - Make an SQS worker timezone aware

294

Problem statement

I have a cron which queues jobs to a worker. The worker sends emails to users. The jobs are queued for different countries. The emails should only be sent between 11 AM to 5 PM for each country i.e. emails to users from Australia should only be sent between 11 AM to 5 PM in Australian time and same goes for all the other countries.

What I've tried

Each of the jobs knows about it's timezone. On reading the job, if it doesn't match with the time requirement, it is deleted from the queue and added with a delay. The delay is calculated based on the difference from the next acceptable time frame to current time.

But the problem with this solution is that the delay is typically in 17-18 hours. Amazon's SQS has maximum delay of 15 minutes. One workaround can be to delay the job by 5 minutes every time but it is inefficient since the same job will be picked several times in those 17-18 hours.

Another terribly inefficient solution can be to have multiple workers for each country.

Presently, I have multiple cron entries for the same command to queue the jobs at the times which are acceptable i.e. jobs for Australia are queued at 11 AM since the cron line for Australian jobs is set to 11 AM.

I am looking for an efficient mechanism to solve this. All ideas are welcome. Can someone please help me out here? Also, what are the standard ways these kind of problems are solved?

481

Answer

Solution:

Instead of deleting the message from the queue, you could instead change its visibility timeout.

This action leaves the message in the "in flight" status waiting for you to delete it (which you won't actually be doing) or until the visibility timeout is reached. When the timeout expires, the message will leave the "in flight" status and immediately become eligible for delivery again... at which point, your worker will receive it again, and can decide whether to act on it, or reset the visibility timeout yet again.

Some limitations and considerations, here:

  • Visibility timeout has a maximum period of 12 hours -- not a serious limitation, since the worst case would be that you'd have to handle it one more time before it was eligible for delivery.

  • If you are using a dead letter queue, the counter for deliveries must be high enough that these operations to change visibility don't send the message to the dead letter queue prematurely, because this will increase the approximate receive count on each message by 1 each time its visibility timeout expires. Also a minor issue.

  • The major consideration: Queues have a limit of 120,000 messages in flight at any one time. Because this action leaves the message in the "in flight" state, this solution breaks down if you expect to leave that many messages waiting for their processing window to arrive. Once you have that many messages in flight, your long polls will fail until some of the messages become visible again, because receiving more messages would increase your in flight counter above the maximum allowed of 120,000. If your volume is much lower than this, the solution should be fine.

Note that I am not suggesting changing the visibility timeout for the queue, but rather sending a request to SQS to change the visibility timeout for each message that you want to defer processing on.

People are also looking for solutions to the problem: php - Uploading symfony project in gitlab

Source

Didn't find the answer?

Our community is visited by hundreds of web development professionals every day. Ask your question and get a quick answer for free.

Ask a Question

Write quick answer

Do you know the answer to this question? Write a quick response to it. With your help, we will make our community stronger.

Similar questions

Find the answer in similar questions on our website.