Automating S3 Events with AWS Lambda

Introduction

Amazon S3 events let you automatically respond to changes in your storage, like when files are uploaded, deleted, or modified. By linking S3 with AWS Lambda, you can create workflows that instantly react to these events without needing to manage servers. This makes it easy to automate tasks like processing uploads, resizing images, or updating records in real-time. In this post, we’ll dive into how you can handle S3 events with AWS Lambda in a simple and efficient way.

Implementing S3 Event Handling with AWS Lambda

To understand what exactly happens when an S3 event is triggered, this post will guide you through the implementation of the architecture below. By examining a practical example, we’ll explore how S3 events are processed and handled using AWS Lambda, providing you with a clear, hands-on understanding of the entire workflow.

The idea is to upload an image to an S3 bucket named original-images. This upload will trigger an S3 event that is intercepted by a Lambda function written in Node.js. The Lambda function processes the incoming event, retrieves the uploaded image, and generates a thumbnail version. This automated process efficiently handles image resizing and stores the resulting thumbnails in a designated thumbnails bucket.

Provision resources using CloudFormation

AWS CloudFormation is a service that helps you automate the setup and management of AWS resources using code. Instead of manually configuring individual services, you define them in a JSON or YAML template. CloudFormation then provisions, updates, or deletes your resources in a consistent and repeatable way, making infrastructure management more efficient and reliable.

Create the CloudFormation template

For me, I prefer to use YAML, but feel free to use JSON if you want. Let's start by creating a file called deployment.yaml.

#deployment.yaml
AWSTemplateFormatVersion: "2010-09-09"
Description: "CloudFormation template to provision resources"

These two lines in the template specify the version of the template format being used and provide a description of the template's purpose.

Now, let's add a Resources section to the template and start by creating the logical resource for the first S3 bucket. This bucket will store the original images and trigger the file creation event.

Note: Resources inside the CloudFormation template are called logical resources. When you create a CloudFormation stack based on the template, a physical resource is created for each logical resource.

Resources:
  # First S3 Bucket
  OriginalImagesBucket:
    Type: "AWS::S3::Bucket"
    Properties:
      BucketName: "mz-original-images-07"
      NotificationConfiguration:
        LambdaConfigurations:
          - Event: "s3:ObjectCreated:*"
            Function: !GetAtt MzFunction.Arn

This snippet defines the mz-original-images-07 in the template:

OriginalImagesBucket: This is the logical name for the S3 bucket resource.
Type: Specifies the AWS resource type, which is AWS::S3::Bucket in this case.
Properties: Contains the configuration details for the S3 bucket.
BucketName: The name of the bucket, which is mz-original-images-07.
NotificationConfiguration: Configures the bucket to send notifications.
LambdaConfigurations: Specifies the Lambda function to be triggered when an object is created in the bucket.
Event: The type of event that triggers the Lambda function, in this case, s3:ObjectCreated:*.
Function: Refers to the ARN of the Lambda function, which is specified as !GetAtt MzFunction.Arn. The Lambda function is named MzFunction.

Defines the mz-thumbnails-07 bucket

# Second S3 Bucket
ThumbnailsBucket:
  Type: "AWS::S3::Bucket"
  Properties:
    BucketName: "mz-thumbnails-07"

Note: S3 bucket names need to be globally unique, which is why I added mz and 07 to the names.

Defines the MzFunction

Before defining the Mz Lambda function itself, we first need to define the execution role that the function will use in order to react to the creation event, process the file, and then store it in the other bucket. The Lambda function needs the proper permissions to perform all of these actions.

# IAM Role for Lambda
LambdaExecutionRole:
  Type: "AWS::IAM::Role"
  Properties:
    RoleName: "LambdaExecutionRole"
    AssumeRolePolicyDocument:
      Version: "2012-10-17"
      Statement:
        - Effect: "Allow"
          Principal:
            Service: "lambda.amazonaws.com"
          Action: "sts:AssumeRole"
    ManagedPolicyArns:
      - "arn:aws:iam::aws:policy/service-role/AWSLambdaBasicExecutionRole"
      - "arn:aws:iam::aws:policy/AmazonS3FullAccess"

This defines an IAM role called LambdaExecutionRole that allows AWS Lambda to assume the role and execute functions. It grants the Lambda function basic execution permissions and full access to S3 through managed policies.

Now, let's jump into defining our MZFunction itself

# Lambda Function
MzFunction:
  Type: "AWS::Lambda::Function"
  Properties:
    FunctionName: "mz-function"
    Handler: "index.handler"
    Role: !GetAtt LambdaExecutionRole.Arn
    Runtime: "nodejs18.x"
    Timeout: 30
    MemorySize: 128
    Code:
      ZipFile: |
        exports.handler = async (event) => {
          console.log("Empty Lambda", event);
          return true;
        }

This AWS CloudFormation code above creates our Lambda function "mz-function" using Node.js 18.x. It runs the index.handler function with a 30-secondtimeout, 128 MB of memory. About the function code, I've added a simple console.log() for now, and we'll update it later.

The last thing we have to do is to give the S3 for our originals image the permission to triggered the Lambda function.

# S3 Bucket Notification for Lambda Trigger
LambdaInvokePermission:
  Type: "AWS::Lambda::Permission"
  Properties:
    FunctionName: !GetAtt MzFunction.Arn
    Action: "lambda:InvokeFunction"
    Principal: "s3.amazonaws.com"
    SourceArn: "arn:aws:s3:::mz-original-images-07"

The above AWS CloudFormation code grants S3 permission to invoke the Lambda function MzFunction when an event occurs. It allows the S3 bucket mz-original-images-07 to trigger the function. The Principal is set to S3, specifying the source of the event.

Deploy the CloudFormation Template

To deploy a CloudFormation template via the AWS Management Console, follow these steps:

Step	Action
1	Sign in to AWS Console: Open the AWS Management Console and navigate to the CloudFormation service.
2	Create Stack: Click on "Create stack" and select "With new resources (standard)."
3	Upload Template: Choose "Upload a template file" and select our deployment.yaml template.
4	Configure Stack: Enter a stack name, configure stack options, and set parameters as required.
5	Review and Create: Review your settings, and click "Create stack" to deploy the template. AWS will start provisioning the resources defined in your template.

To visualize the template's logical resources, simply click the View in Application Composer button.

After a few minutes, our resources will be created successfully

Test if the Lambda function is triggered when an image is uploaded

To view the console.log output of our Lambda function, follow these steps:

Step	Action
1	Upload an image to the S3 Originals Images bucket.
2	Go to the Lambda service.
3	Select our Lambda function: Choose the Lambda function whose logs we want to view.
4	Go to the Monitoring tab: Click on the Monitoring tab for the selected Lambda function. Under the Logs section, click on View logs in CloudWatch. This will open the CloudWatch Logs console.
5	View logs in CloudWatch: In the CloudWatch Logs console, look for the log group with the name /aws/lambda/mz-function. Click on the log group to view the log streams. Select the most recent log stream to see the output of our console.log statements.

mz-function console.log output

mz-function configs

Add the lambda function code

To add the Lambda code, follow these steps in the AWS Management Console:

Navigate to the Code tab for your Lambda function.
Enter the following code:

const s3 = new (require("aws-sdk").S3)();

// Define the source and destination buckets
const SOURCE_BUCKET = "mz-original-images-07"; // Original images bucket
const THUMBNAILS_BUCKET = "mz-thumbnails-07"; // Thumbnails bucket

exports.handler = async (event) => {
  // Extract S3 object information from the event
  const record = event.Records[0];
  const bucket = record.s3.bucket.name;
  const key = decodeURIComponent(record.s3.object.key.replace(/\+/g, " "));

  // Check if the bucket is the source bucket
  if (bucket !== SOURCE_BUCKET) {
    console.log("Not an image upload event for the source bucket.");
    return;
  }

  try {
    // Get the image from S3
    const params = {
      Bucket: bucket,
      Key: key,
    };
    const { Body } = await s3.getObject(params).promise();

    // Process the image (Here, we just re-upload the original image for demonstration)
    // To resize, you would typically need a library, but we'll skip resizing for simplicity

    // Define the destination key
    const thumbnailKey = `thumbnails/${key}`;

    // Upload the image to the thumbnails bucket
    const uploadParams = {
      Bucket: THUMBNAILS_BUCKET,
      Key: thumbnailKey,
      Body: Body,
      ContentType: "image/jpeg",
    };
    await s3.putObject(uploadParams).promise();

    console.log(`Image uploaded to ${THUMBNAILS_BUCKET}/${thumbnailKey}`);
  } catch (error) {
    console.error("Error processing the image:", error);
    throw error;
  }
};

the code is about extracting the image from an S3 bucket, checks if it's from the source bucket, and then re-uploads it to a different S3 bucket.

Note 1 : The image is simply copied without resizing for demonstration purposes.

Note 2 : If you encounter the error "Runtime.ImportModuleError: Error: Cannot find module 'aws-sdk'" whenever your Lambda function is triggered, try downgrading your Node.js runtime from the AWS Management Console. This issue is known to occur with the Node.js 18 runtime, and switching to an earlier version may resolve it.

Finally : click the Deploy button to apply the changes to our Lambda code.

Conclusion

In this blog, we demonstrated how to deploy a CloudFormation template and configure an AWS Lambda function to respond to S3 events. The Lambda function is triggered when an image is uploaded to an S3 bucket, allowing for automated processing or handling of the image. This setup illustrates the basics of integrating S3 with Lambda to streamline workflows in a serverless environment.