AWS SQS Jobs Processor

A reference project to deploy a serverless jobs processor on AWS with Terraform

A service which returns Fibonacci sequence

App URL here: https://8nalvgvzrl.execute-api.us-east-1.amazonaws.com/production/api/v1/jobs

AWS Architecture

Pre-requisite

Make sure you have installed Terraform, AWS CLI, and configured a default AWS CLI profile (see doc here)

terraform -help # prints Terraform options
which aws # prints /usr/local/bin/aws
aws --version # prints aws-cli/2.0.36 Python/3.7.4 Darwin/18.7.0 botocore/2.0.0
aws configure # configure your AWS CLI profile

Configuration

Create an S3 bucket to store Terraform state. Populate bucket name in 01-main.tf

Create another S3 bucket to store Lambda functions build artifacts:

aws s3api create-bucket --bucket=<YOURUNIQUEBUCKETNAME> --region=<DEFAULTREGION>

Populate terraform.tfvars:

defaultregion      = "<YOURAWSDEFAULTREGION>"
appname            = "<GIVEYOURAPPA_NAME!>"
environment         = "<ENVIRONMENT_NAME>"

Deploy Lambda package

Navigate to /deploy/lambdas/<FUNCTION_NAME>
Create a .zip file:

zip -r <FUNCTION_NAME>.zip .

Uploads Lambda artifact to S3 bucket:

aws s3 cp <FUNCTIONNAME>.zip s3://<BUCKETNAME>/v1.0.0/<FUNCTION_NAME>.zip

Deploy Lambda layer

cd deploy/lambdas/layer # change to lambda layer directory
sh updateLayer.sh # uploads lambda layer to S3 bucket

Deploy AWS stack

cd deploy # change to deploy directory
terraform init # initialises Terraform
terraform apply # deploys AWS stack. See output for API url
terraform destroy # destroys AWS stack

Usages

Make a POST request to <API_ENDPOINT>/api/v1/jobs to trigger a job which returns Fibonacci sequence with N numbers

{
  "data": 10
}

Make a GET request to <API_ENDPOINT>/api/v1/jobs to retrieve all jobs
Make a GET request to <APIENDPOINT>/api/v1/jobs/<JOBID> to retrieve a specific job

Update Lambda package

Update the deploy/lambdas/processQueue/updateFunction.sh shell script with correct values i.e. Amazon S3 bucket name, and key. See documentation on AWS CLI update-function-code here

Run the script:

cd deploy/lambdas/processQueue # change to lambda directory
chmod +x updateFunction.sh # set permission to run script
./updateFunction.sh # run the script