Serverless computing with AWS Lambda, Part 2: AWS Lambda with DynamoDB

Integrate AWS Lambda with DynamoDB, then call Lambda functions from a Java client

Rudolf Getel

The first half of this article presented an overview of serverless computing with AWS Lambda, including building, deploying, and testing AWS Lambda functions in an example Java application. In Part 2, you'll learn how to integrate Lambda functions with an external database, in this case DynamoDB. We'll then use the AWS SDK to invoke Lambda functions from our example Java application.

AWS Lambda and DynamoDB

DynamoDB is a NoSQL document store that is hosted by Amazon Web Services (AWS). DynamoDB defines data abstractions as tables, which accept common database operations such as insert, retrieve, query, update, and delete. As with many other NoSQL databases, DynamoDB's schema isn't fixed, so some items in the same table can have fields that others do not.

One of DynamoDB's best features is its tiered pricing model. Unlike the AWS Relational Database Service (RDS), in which AWS manages your database using EC2 instances that you pay for, DynamoDB is pay-as-you-go. You pay for the storage you use and the throughput of your queries, but you don't directly pay for any underlying virtual machines. Additionally, AWS gives you a free tier supporting up to 25 GB of space, with enough throughput to execute up to 200 million requests per month.

In Serverless computing with AWS Lambda, Part 1, we developed a simple, serverless Java application using Lambda functions. You can download the source code for the GetWidgetHandler application anytime. If you haven't already read Part 1, I suggest familiarizing yourself with the application code and examples from that article before proceeding.

Our first step is to setup the DynamoDB database in our AWS console. After that we'll update the get-widget function from Part 1 to retrieve a widget from a DynamoDB table.

Setup the DynamoDB database in AWS

We'll start by creating the DynamoDB table. From the AWS console, click on Services and choose DynamoDB from the database section, as shown in Figure 1.

Steven Haines

Figure 1. Launch the DynamoDB dashboard

Once launched, you'll see the DynamoDB dashboard. Click the Create table button to start creating your table, shown in Figure 2.

Steven Haines

Figure 2. Create a DynamoDB table

Now you'll see the page shown in Figure 3.

Steven Haines

Figure 3. Create a DynamoDB table

Give your table a name (in this case "Widget") and set the primary key to id, leaving it as a String. Pressing Create when you are finished will direct you to the DynamoDB tables page. If you need to navigate to this page in the future, select Services-->DynamoDB, and click on Tables.

Steven Haines

Figure 4. DynamoDB tables page

We'll manually create an entry in the new Widget table, so click the Create item button shown in Figure 5.

Steven Haines

Figure 5. Create item page

DynamoDB will pre-populate the Create Item page with the id field. Enter an ID that is easy to remember, such as "1". Next, press the plus (+) next to the new ID, adding another field called name. Enter a value for the name field, such as "Widget 1". Press Save when you are finished.

Update the GetWidgetHandler class

With data in our database, the next thing we need to do is update the GetWidgetHandler class from Part 1. We'll start by adding the DynamoDB dependency to our original POM file. The updated pom.xml file is shown in Listing 1.

Listing 1. pom.xml (updated with DynamoDB dependency)

<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
         xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 http://maven.apache.org/maven-v4_0_0.xsd">
    <modelVersion>4.0.0</modelVersion>
    <groupId>com.javaworld.geekcap</groupId>
    <artifactId>aws-lambda-java</artifactId>
    <packaging>jar</packaging>
    <version>1.0-SNAPSHOT</version>
    <name>aws-lambda-java</name>
    <url>http://maven.apache.org</url>

    <properties>
        <java.version>1.8</java.version>
        <project.build.sourceEncoding>UTF-8</project.build.sourceEncoding>
    </properties>


    <dependencies>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-lambda-java-core</artifactId>
            <version>1.1.0</version>
        </dependency>
        <dependency>
            <groupId>com.amazonaws</groupId>
            <artifactId>aws-java-sdk-dynamodb</artifactId>
            <version>1.11.135</version>
        </dependency>
        <dependency>
            <groupId>junit</groupId>
            <artifactId>junit</artifactId>
            <version>4.12</version>
            <scope>test</scope>
        </dependency>
    </dependencies>

    <build>
        <plugins>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-compiler-plugin</artifactId>
                <version>2.0.2</version>
                <configuration>
                    <source>${java.version}</source>
                    <target>${java.version}</target>
                </configuration>
            </plugin>
            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-shade-plugin</artifactId>
                <version>2.3</version>
                <configuration>
                    <createDependencyReducedPom>false</createDependencyReducedPom>
                </configuration>
                <executions>
                    <execution>
                        <phase>package</phase>
                        <goals>
                            <goal>shade</goal>
                        </goals>
                    </execution>
                </executions>
            </plugin>
        </plugins>
    </build>
</project>

Listing 1 adds the aws-java-sdk-dynamodb dependency to the POM file from Part 1. Listing 2 shows the updated GetWidgetHandler class.

Listing 2. GetWidgetHandler.java (updated to load data from DynamoDB)


package com.javaworld.awslambda.widget.handlers;

import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
import com.amazonaws.services.dynamodbv2.document.DynamoDB;
import com.amazonaws.services.dynamodbv2.document.Item;
import com.amazonaws.services.dynamodbv2.document.Table;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.javaworld.awslambda.widget.model.Widget;
import com.javaworld.awslambda.widget.model.WidgetRequest;

public class GetWidgetHandler implements RequestHandler<WidgetRequest, Widget> {
    @Override
    public Widget handleRequest(WidgetRequest widgetRequest, Context context) {
        //return new Widget(widgetRequest.getId(), "My Widget " + widgetRequest.getId());

        // Create a connection to DynamoDB
        AmazonDynamoDB client = AmazonDynamoDBClientBuilder.defaultClient();
        DynamoDB dynamoDB = new DynamoDB(client);

        // Get a reference to the Widget table
        Table table = dynamoDB.getTable("Widget");

        // Get our item by ID
        Item item = table.getItem("id", widgetRequest.getId());
        if(item != null) {
            System.out.println(item.toJSONPretty());

            // Return a new Widget object
            return new Widget(widgetRequest.getId(), item.getString("name"));
        }
        else {
            return new Widget();
        }
    }
}

The main interface to DynamoDB is the DynamoDB object. In order to create a DynamoDB instance, we need a DynamoDB client. Because our Lambda function will run in AWS, we do not need to provide credentials, so we can use the default client. Note that we'll only be able to query the database without credentials because the get-widget-role from Part 1 has the dynamodb:GetItem permission.

From the DynamoDB instance, we can call getTable("Widget") to retrieve a Table instance. Then we can call getItem() on the Table instance, passing it the primary key of the item we want to retrieve. If there is an item with the specified primary key then it will return a valid response; otherwise it will return null. The Item class provides access to the response parameters, so we finish up the implementation by creating a new Widget object with its name loaded from DynamoDB.

download
Get the code for the updated GetWidgetHandler application. Created by Steven Haines for JavaWorld.

Querying DynamoDB with DynamoDBMapper

There are several APIs for querying DynamoDB, from a RESTful service call, to the low-level interface above, to a couple of higher level interfaces. One of the more popular interfaces is DynamoDBMapper. This interface provides a similar construct to what you might find when mapping objects to relational data in a tool like Hibernate. Let's briefly review how to retrieve a Widget from DynamoDB using the DynamoDBMapper API.

The first thing that we need to do is add a few annotations to the Widget class, which is shown in Listing 3.

Listing 3. Widget.java (updated with DynamoDBMapper annotations)


package com.javaworld.awslambda.widget.model;

import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBAttribute;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBHashKey;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBTable;

@DynamoDBTable(tableName="Widget")
public class Widget {
    private String id;
    private String name;

    public Widget() {
    }

    public Widget(String id) {
        this.id = id;
    }

    public Widget(String id, String name) {
        this.id = id;
        this.name = name;
    }

    @DynamoDBHashKey(attributeName="id")
    public String getId() {
        return id;
    }

    public void setId(String id) {
        this.id = id;
    }

    @DynamoDBAttribute(attributeName="name")
    public String getName() {
        return name;
    }

    public void setName(String name) {
        this.name = name;
    }
}

The DynamoDBTable annotation specifies the name of the DynamoDB table to which the Widget maps. The DynamoDBHashKey annotation identifies the primary key of the Widget table. And the DynamoDBAttribute annotation identifies other class attributes that map to database attributes in DynamoDB. If you had other attributes that you wanted to ignore, you could add the @DynamoDBIgnore annotation.

With the Widget class annotated, we can now update the GetWidgetHandler class to use the DynamoDBMapper, which is shown in Listing 4.

Listing 4. GetWidgetHandler.java (updated with DynamoDBMapper)


package com.javaworld.awslambda.widget.handlers;

import com.amazonaws.services.dynamodbv2.AmazonDynamoDB;
import com.amazonaws.services.dynamodbv2.AmazonDynamoDBClientBuilder;
import com.amazonaws.services.dynamodbv2.datamodeling.DynamoDBMapper;
import com.amazonaws.services.lambda.runtime.Context;
import com.amazonaws.services.lambda.runtime.RequestHandler;
import com.javaworld.awslambda.widget.model.Widget;
import com.javaworld.awslambda.widget.model.WidgetRequest;

public class GetWidgetHandler implements RequestHandler<WidgetRequest, Widget> {
    @Override
    public Widget handleRequest(WidgetRequest widgetRequest, Context context) {
        // Create a connection to DynamoDB
        AmazonDynamoDB client = AmazonDynamoDBClientBuilder.defaultClient();

        // Build a mapper
        DynamoDBMapper mapper = new DynamoDBMapper(client);

        // Load the widget by ID
        Widget widget = mapper.load(Widget.class, widgetRequest.getId());
        if(widget == null) {
            // We did not find a widget with this ID, so return an empty Widget
            context.getLogger().log("No Widget found with ID: " + widgetRequest.getId() + "\n");
            return new Widget();
        }
        
        // Return the widget
        return widget;
    }
}

In the former (Part 1) version of the GetWidgetHandler we created an AmazonDynamoDB instance, using a AmazonDynamoDBClientBuilder.defaultClient() call. Now we'll use that client to initialize a DynamoDBMapper instance instead.

The DynamoDBMapper class provides access to execute queries, load objects by ID, save objects, delete objects, and so forth. In this case, we pass DynamoDBMapper the widget's class (Widget.class) and its primary key. If DynamoDB has a Widget with the specified primary key it will return it; if not it will return null.

Rebuild and then re-upload your new JAR file by opening your Lambda function dashboard, then click on the Code tab and press Upload. When you re-upload and subsequently call your function, AWS Lambda will create a new container for the new JAR file and push that to an EC2 instance. You should expect the first run to be slow.

If you happen to encounter an OutOfMemoryError when you re-test your function, select the Configuration tab and open the Advanced Settings section. Here you can increase your memory, as shown below.

Steven Haines

Figure 6. Changing memory settings

Calling AWS Lambda functions from a Java application

Now that we have a function running in AWS Lambda, we'll write a client application in Java that can call it. In order to execute a Lambda function from a Java application that isn't running in AWS, we first need to create an IAM user with permissions to invoke the Lambda function. We can then use those credentials from our client application.

All of this is easy conceptually, but there are a lot of steps in the AWS console. I'll walk you through them with screenshots.

Step 1. Create the AWS user

We don't want to embed our primary user credentials into a Java application, so we're going to create a new user with fewer privileges. Navigate to the AWS console, choose Services, and find IAM under Security, Identity & Compliance. Click on Users and then the Add user button, as shown below.

Steven Haines

Figure 7. Add user

Give your user a name, such as get-widget-lambda-user and check the Programmatic access checkbox and press Next: Permissions, as shown below.

Steven Haines

Figure 8. Set user access

Next, click on the Create Group button.

Steven Haines

Figure 9. Create a group for your user

While we can add inline policies to users, it is a better practice to create a group that manages policies for you, and then add the user to that group. Enter a name for your group, then click Create policy.

Steven Haines

Figure 10. Create a group for your user

Step 2. Create a custom access policy

We could create a manual policy but to try to make things easier I searched the existing policies first. Unfortunately, the only existing policy for granting lambda:InvokeFunction permissions is the Lambda All Access policy. This basically gives the user root access to all Lambda functionality, which is not a good idea.

We'll need to create a custom policy to restrict access for a programatic user. First, we'll build a policy using the policy generator. To start, click on Create policy. You should see a screen like this one:

Steven Haines

Figure 11. Create policy

Click Select at the end of the Policy Generator line. This will bring up the Edit Permissions page shown here:

Steven Haines

Figure 12. Edit Permissions

Choose AWS Lambda from the AWS Service dropdown, then click on Actions and choose InvokeFunction. Add the following for the ARN:

arn:aws:lambda:*

This policy allows the InvokeFunction action to be performed on any AWS Lambda function. You could further restrict this policy to limit invocation to a single AWS Lambda by copying the ARN for the AWS Lambda itself, which is on the top of the Lambda's dashboard page. Mine is:

arn:aws:lambda:us-east-1:YOUR_ACCOUNT_NUMBER:function:get-widget

When you are ready, press Add Statement to add this permission to your policy. Press Next Step to review the policy, as shown here.

Steven Haines

Figure 13. Review policy

The important thing on this page is to set a policy name, otherwise your policy will be named "policygen" followed by some random numbers. Press the Create Policy button. Because you had to create the policy in a new window, go back to the user-creation screen, press the Refresh button, and find and select your policy, as shown:

Steven Haines

Figure 14. Refresh the Create Group page

If it's difficult to find your newly created policy, you can use filters to help. As an example, try selecting customer managed policies and enter the name "lambda-invoke". Press Create Group and it will return you to your user-creation workflow with your new group:

Steven Haines

Figure 15. Select group

Finally, review your user and press Create User:

Steven Haines

Figure 16. Review user

The last page is important because this is where you can see your access key ID and secret access key. If you press the Show link, it will reveal your secret access key. You'll need to add this information to your Java code in the next section, so be sure to preserve it.

Steven Haines

Figure 17. New user

That completes the policy setup. So far you have

  1. Created a new policy allowing the invocation of any Lambda function in your account.
  2. Created a group that has this single policy.
  3. Created a user and added it to this group.
  4. Saved your access key ID and secret access key, which you'll add to your Java code in the next section.

Step 3. Create a Lambda client class

Next we'll write the code to connect to AWS and invoke our function. Listing 3 shows the source code for the WidgetLambdaClient class.

Listing 3. WidgetLambdaClient.java


package com.javaworld.awslambda.widget.client;

import com.amazonaws.auth.AWSStaticCredentialsProvider;
import com.amazonaws.auth.BasicAWSCredentials;
import com.amazonaws.regions.Regions;
import com.amazonaws.services.lambda.AWSLambda;
import com.amazonaws.services.lambda.AWSLambdaClientBuilder;
import com.amazonaws.services.lambda.model.InvokeRequest;
import com.amazonaws.services.lambda.model.InvokeResult;

public class WidgetLambdaClient {

    public static void main(String[] args) {
        // Setup credentials
        BasicAWSCredentials awsCreds = new BasicAWSCredentials("YOUR_ACCESS_KEY_ID", "YOUR_SECRET_ACCESS_KEY");

        // Create an AWSLambda client
        AWSLambda lambda = AWSLambdaClientBuilder.standard().withCredentials(new AWSStaticCredentialsProvider(awsCreds)).withRegion(Regions.US_EAST_1).build();

        // Create an InvokeRequest
        InvokeRequest request = new InvokeRequest()
                .withFunctionName("get-widget")
                .withPayload("{ \"id\": \"1\"}");

        try {
            // Execute the InvokeRequest
            InvokeResult result = lambda.invoke(request);

            // We should validate the response
            System.out.println("Status Code: " + result.getStatusCode());

            // Get the response as JSON
            String json = new String(result.getPayload().array(), "UTF-8");

            // Show the response; we could use a library like Jackson to convert this to an object
            System.out.println(json);
        } catch(Exception e) {
            e.printStackTrace();
        }
    }
}

We begin by creating a BasicAWSCredentials instance with our access key and secret key, which you preserved in Step 2. Our main interface into AWS Lambda is the AWSLambda class, which can be created using the AWSLambdaClientBuilder. We invoke standard() to create it and then we add our user credentials, which we've wrapped in an AWSStaticCredentialsProvider instance.

Lambda functions are created in specific regions, so you'll need to set the region. If you aren't sure where you created your Lambda function, navigate back to your Lambda page and look at the ARN in the upper-right side of the page. Or you can look at the upper-right side of the top toolbar next to your name on the Lambda page in the AWS console. As an example, I am running in "N. Virginia", which is "us-east-1". Use this website to translate the physical location to the region. Once you've set the region, call build() to create the AWSLambda instance.

Step 4. Invoke the Lambda function

There are different ways of invoking Lambdas. In this case we'll opt for the most straightforward (and manual) way, of sending an InvokeRequest to the function and receiving an InvokeResult back. To start, create a new InvokeRequest instance, call withFunction() to tell it the name of the function you want to invoke, and then pass the payload you want to send via the withPayload() method. This payload should look just like the one we used for testing in the AWS Lambda dashboard.

We pass the InvokeRequest to AWSLambda's invoke() method and it returns an InvokeResult instance. The getStatusCode() method will tell us if it succeeded (returning a 200 response) or failed (returning a non-2xx response.) In a production application you should examine the status code after every Lambda invocation and respond accordingly.

We can retrieve the body of the response by calling the InvokeResult's getPayload() method, which returns a ByteBuffer. We can convert this to a raw JSON string by converting the ByteBuffer to an array, by calling its array() method, and then passing that array to the String's constructor. If we wanted to convert this into an object, we could use a tool like Jackson or Gson to deserialize the JSON into an object.

Make an executable JAR file

Finally, to make running this code easier, we'll add the maven-jar-plugin to the POM file, referencing our WidgetLambdaClient in the mainClass:

Listing 8. Add the maven-jar plugin to the Maven POM file


            <plugin>
                <groupId>org.apache.maven.plugins</groupId>
                <artifactId>maven-jar-plugin</artifactId>
                <configuration>
                    <archive>
                        <manifest>
                            <addClasspath>true</addClasspath>
                            <classpathPrefix>lib/</classpathPrefix>
                            <mainClass>com.javaworld.awslambda.widget.client.WidgetLambdaClient</mainClass>
                        </manifest>
                    </archive>
                </configuration>
            </plugin>

Step 5. Build and monitor the Lambda client class

Build with mvn clean install and then you can execute from the target directory with the following command:

java -jar aws-lambda-java-1.0-SNAPSHOT.jar

When I run this code, I see the following output:

Status Code: 200
{"id":"1","name":"Widget 1"}

If you run into problems, check that you have the proper credentials configured and that your IAM user has the proper permissions to execute your function. You could also check the AWS Lambda logs, which you may access either through the Lambda dashboard or directly from CloudWatch:

  1. From your Lambda page, click on the Monitoring tab and then choose View Logs in CloudWatch in the upper-right corner
  2. From the Services menu, choose CloudWatch, under Management Tools, as shown in Figure 18. Select Logs from the left panel and then choose the aws/lambda/get-widget log group.
Steven Haines

Figure 18. Opening CloudWatch

Accessing logs through your Lambda page will automatically filter to your Lambda's filter group, but regardless of how you got there, you should see something similar to Figure 19.

Steven Haines

Figure 19. CloudWatch logs for your Lambda's log group

Each Log Stream represents the periodic rollup of logs for a specific Lamdba. You can click on one of the entries to see the contents of the log; for example, I updated the client to request the Widget with ID "2", which does not exist. Figure 20 shows the contents of the logs for that execution.

Steven Haines

Figure 20. CloudWatch logs showing an error

In this example you can see a successful execution (start and end without any custom logging) followed by an error ("No widget found with ID: 2"). Logs are a good tool to help you troubleshoot Lambdas, especially ones executed asynchronously.

Conclusion

This two-part tutorial has introduced you to serverless computing with AWS Lambda. In Part 1 we answered the question, "What is serverless computing, anyway?" and I explained the relationship between serverless computing, microservices, and nanoservices architectures. You got your first look at AWS Lambda and we built, deployed, and tested our first Lambda function in Java.

In Part 2, we've added support for Amazon's DynamoDB, enabling our Lambda function to retrieve a live Widget from DynamoDB instead of creating one on-the-fly. You learned how to interact with DynamoDB using the DynamoDBMapper API. We created a Java client application that could invoke our Lambda function, and you learned how to create a new IAM user, group, and custom policy. Finally, we built the updated Lambda function and reviewed logs captured and sent to CloudWatch for troubleshooting.