Background Process in Lambda Using SQS

AWS Lambda

AWS Lambda is a serverless, event-driven compute service that lets you run code for virtually any type of application or backend service without provisioning or managing servers. You can trigger Lambda from over 200 AWS services and software as a service (SaaS) applications and only pay for what you use.

The Problem

We want some work to process at background (asynchronously) in Lambda even if the Lambda function returns. But when the lambda function returns, all the background services automatically stop.

The Solution

So here we are going to use Amazon SQS for this purpose. We will send a message to a queue, and it will trigger another lambda function that will run in the background.

What is Amazon SQS?

Amazon Simple Queue Service (SQS) is a managed message queuing service that enables you to decouple and scale microservices, distributed systems, and serverless applications. Using SQS, you can send, store, and receive messages asynchronously between software components at any volume, without losing messages or requiring other services to be available.

SQS offers two types of message queues. Standard queues offer maximum throughput, best-effort ordering, and at-least-once delivery. SQS FIFO queues are designed to guarantee that messages are processed exactly once, in the exact order that they are sent.

Golang Example to Run Background Processes in Lambda Using SQS

Prerequisites

• For building and deploying your functions, you’ll be using the Serverless Framework, which is the most widely used tool for the job. Assuming you have a recent version of Node.js installed, you can install the Serverless CLI with the following npm command

  npm install -g serverless
  

  Once you have the Serverless CLI installed, you must configure it to use the AWS access keys of your account

  serverless config credentials --provider aws --key <access key ID> --secret <secret access key>
  

  You can get the access key and secret key from the My Security Credentials option. Use Create New Access Key if you do not have one already

  

• If you don’t have Go installed yet, you can either download an installer from the official website or use your favorite package manager to install it

Create a Messaging Queue

Let’s create a queue. Go to Simple Queue Service (SQS) in your AWS account. You will see the following interface there

Create queue from here and select type FIFO and let’s name it test.fifo as we want to process exactly once message, in the exact order that they are sent. you can also use the standard queue according to your need. And we’ll keep all other settings to default for now.

And now let’s code.

Sending Message on the SQS Queue

Now that you have everything you need, let’s install the AWS SDK for Go library.

go get github.com/aws/aws-sdk-go

After this, we can proceed to write the code to send a message on our SQS queue to trigger the lambda function that will perform the background task.

package handlers

import (
  "github.com/aws/aws-sdk-go/aws"
  "github.com/aws/aws-sdk-go/aws/credentials"
  "github.com/aws/aws-sdk-go/aws/session"
  "github.com/aws/aws-sdk-go/service/sqs"
  "os"

  "encoding/json"
  "log"
  "net/http"
)


func yourHttpHandlerFunction(w http.ResponseWrite, r *http.Request) {
  // tasks you need to perform before the background task
  ...
  
  // sending details to other lambda function to perform the background task
  message := SqsTriggerMessage{
    Message: "You can add different fields here according to your data requirements in the background task.",
  }

  err := sendMessage(message)
  if err != nil {
    log.Println(err)
 w.WriteHeader(http.StatusInternalServerError)
    return
  }

  // other tasks you need to perform
  ...
  
  // sending user an immediate response
  w.WriteHeader(http.StatusOK)
}

type SqsTriggerMessage struct {
  Message string
}

func sendMessage(data interface{}) error {
  b, err := json.Marshal(data)
  if err != nil {
    return err
  }

  svc := sqs.New(
    session.Must(
      session.NewSession(
        &aws.Config{
          Credentials: credentials.NewStaticCredentials(os.Getenv("AWS_ACCESS_KEY_ID"), os.Getenv("AWS_SECRET_ACCESS_KEY"), ""),
          Region:      aws.String(os.Getenv("AWS_REGION")),
        },
      ),
    ),
  )

  result, err := svc.GetQueueUrl(&sqs.GetQueueUrlInput{
    QueueName: aws.String("queue-name"),
  })
  if err != nil {
    return err
  }

  _, err = svc.SendMessage(&sqs.SendMessageInput{
    MessageBody:            aws.String(string(b)),
    QueueUrl:               result.QueueUrl,
    MessageGroupId:         aws.String("group-id"),
    MessageDeduplicationId: aws.String("deduplication-id"),
  })

  return err
}

This previous code can be broken into a few simple steps:

• Define a message struct that contains the data you need in the background task
• Marshal your struct into JSON and send that JSON as message to the SQS queue We can also send this through the message attributes, but I found this method more convenient to use
• Give the user a 200 response and return. Your lambda function will finish after this, but the message you sent on the SQS queue will trigger the other lambda function to perform the background task

The Lambda Function to Handle SQS Event

Let’s create another program to handle sqs events.

package main

import (
  "github.com/aws/aws-lambda-go/events"
  "github.com/aws/aws-lambda-go/lambda"

  "encoding/json"
  "log"
)

func main() {
  lambda.Start(handleSqsRequest)
}

func handleSqsRequest(sqsEvent events.SQSEvent) error {
  for _, message := range sqsEvent.Records {
    var request SqsTriggerMessage
    err := json.Unmarshal([]byte(message.Body),&request)
    if err!=nil {
      log.Println(err)
      continue
    }
 // task you need to perform based on the data in the message
    log.Println(request.Message)
 ...
  }

  return nil
}

type SqsTriggerMessage struct {
  Message string
}

This previous code can be broken into a few simple steps:

• We wrote a handleSqsRequest function that will receive the messages from queue and registered it in the main function using the AWS Lambda for Go library
• In the event handler function, we are unmarshalling the message body into our message struct, and printing the message on the console. You can send the details of the task you need to perform and use those to call appropriate function to do that task

Now we have 2 Lambda functions the first one sends the message to SQS queue, that needs to trigger our second lambda function to perform the background task

Deployment

Our lambda functions are now ready, and we can proceed by deploying it with the Serverless Framework. Our application is deployed by the Serverless framework based on the serverless.yml configuration file.

If you are not familiar with the .yml syntax, you can read this serverless.yml guide.

We must first create a serverless.yml file that defines what we are deploying.

service: your-service-name
provider:
  name: aws
  runtime: go1.x
package:
  exclude:
    - ./**
  include:
    - ./bin/**
functions:
  main-program:
    handler: bin/main-program
    events:
      - http:
          path: /
          method: get
      ...
  sqs-handler:
    handler: bin/sqs-handler
    events:
      - sqs: <replace_this_with_your_sqs_queue_arn>

In the above file, we defined both our lambda functions.

• The first one runs the main program on the HTTP requests through Amazon API Gateway. In this program we need to perform the background task, so we will send a message on the SQS queue
• The second one runs our sqs handler on an SQS event in the queue which arn we provide here. You can get the arn from your aws account where we created it. Amazon SQS > Queues > test.fifo

Next up, we will build our code, and deploy it using the serverless deploy command.

GOOS=linux GOARCH=amd64 go build -o bin/main-program .
GOOS=linux GOARCH=amd64 go build -o bin/sqs-handler ./sqs

serverless deploy

You can see the logs of all your lambda functions through the CloudWatch. To access the logs, go to CloudWatch > Log groups in your amazon account.

You can see the logs from here. Let’s check the sqs-handler logs. It must have printed the received message.

References

• Amazon SQS
• AWS Lambda and SQS
• AWS Lambda Function Handler in Go
• Serverless Framework and SQS

Get the full source code from this github repository.