We'll explore how to create and execute a map and child context operations

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK – Part 1 Introduction to the series and services

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK – Part 2 Durable synchronous and asynchronous steps

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK – Part 3 Waits and callbacks

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK – Part 4 Wait for condition

Orchestrating workflows with Amazon S3 Files and AWS Lambda durable functions using Java SDK – Part 5 Map and child context

Introduction

In part 2 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to create and execute durable steps synchronously and asynchronously. Later, in parts 3 and 4 of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement wait for callback and wait for condition.

In this part, we’ll explore map and child context operations.

Sample application with map and child context

Let’s explore the child context operation first. Child contexts run an isolated stream of work with their own operation counter and checkpoint log. They support the full range of durable operations — step, wait, invoke, createCallback, and nested child contexts. In our sample application, we’ll add a map operation, which uses a child context underneath. We get access to it through DurableContext. A Map operation applies a function to each item in a collection concurrently, with each item running in its own child context.

Now let’s add an artificial use case: for each YouTube video found, we add it to the author’s YouTube playlist. I first copied the application that we created in part 2 into the aws-s3-files-lambda-durable-functions-with-map-and-child-context-java-25 repository.

The main business logic we have to add is in the saveYouTubeVideosToPlayList method of the AbstractAuthorContentExtractor:

public default void saveYouTubeVideosToPlayList (DurableContext ctx, Author author, YouTubeVideos youtubeVideos)  {
   var youtubePlayList = findOrCreateYouTubePlayListForTheAuthor(author);
   
   var mapConfig = MapConfig.builder()
       .nestingType(NestingType.NESTED)
       .maxConcurrency(5)                                   
       .completionConfig(CompletionConfig.allCompleted())    
       .build();
		
   var mapResult = ctx.map("saveYouTubeVideoToPlayList-step", youtubeVideos.youtubeVideos(), 
      Boolean.class, (youtubeVideo, index, childCtx) -> {		
        return childCtx.step("saveYouTubeVideoToPlayList-step-"+index, Boolean.class, 
       stepCtx -> saveYouTubeVideoToPlayList(youtubeVideo, youtubePlayList)
      , this.getStepConfig());
    }, mapConfig);
		
}
	
	
private static int findOrCreateYouTubePlayListForTheAuthor(Author author) {
     // Use YouTube API to log in to the YouTube Account and search for the author's playlist
     // and return it if it already exists or create a new one
     // move this logic into a separate Lambda function and invoke it as a durable step
     return 1;
}
	
private static Boolean saveYouTubeVideoToPlayList(YouTubeVideo youtubeVideo, int playListId) {
    //Use YouTube API to save the video URL to the playlist. 
    return true;
}

We invoke this saveYouTubeVideosToPlayList method in the AuthorContentExtractor or AsyncAuthorContentExtractor classes.

First of all, I won’t use the YouTube API to create or retrieve the YouTube playlist for the given author and store each found video there. I use some static output, as the goal of this article is to demonstrate the functionality of the map operation.

Let’s go step by step through this code. Map applies a function to each item in a collection concurrently, with each item running in its own child context. Results are collected into a MapResult that maintains input order. Each item’s function receives its own DurableContext, so you can use any durable operation (step(), wait(), invoke(), etc.) inside the map function. The function passed to map() is a MapFunction<I, O>:

@FunctionalInterface
public interface MapFunction<I, O> {
    O apply(I item, int index, DurableContext context);
}

The index parameter is the zero-based position of the item in the input collection, useful for naming operations or correlating results. For each found YouTube video, we invoke the saveYouTubeVideoToPlayList method, which returns the result of type Boolean. In our case, it is always true, indicating that the video was stored in the playlist.

The input collection must have deterministic iteration order. List, LinkedList, and TreeSet are accepted. HashSet and unordered map views are not, leading to the IllegalArgumentException. That’s why I changed the implementation of the YouTubeVideos class to use List instead of Set there.

Next, let’s explore MapConfig in more detail. There are several main settings there:

maxConcurrency controls how many items execute concurrently. When set, items beyond the limit are queued and started as earlier items complete. The default is null (unlimited).

nestingType, with Nested as the default.

Nested. Create CONTEXT operations for each branch/iteration with full checkpointing. Operations within each branch/iteration are wrapped in their own context. Observability: High – each branch/iteration appears as a separate operation in execution history. Cost: Higher – consumes more operations due to CONTEXT creation overhead. Scale: Lower maximum iterations due to operation limits
Flat. Skip CONTEXT operations for branches/iterations using virtual contexts. Operations execute directly without individual context wrapping. Observability: Lower – branches/iterations don’t appear as separate operations. Cost: ~30% lower – reduces operation consumption by skipping CONTEXT overhead. Scale: Higher maximum iterations possible within operation limits

CompletionConfig controls when the map operation stops starting new items. Here, there are many possibilities from which we can choose:

allCompleted (we use it). All items run regardless of failures. Failures are captured per-item.
allSuccessful. All items must succeed. Zero failures tolerated.
firstSuccessful. Complete as soon as the first item succeeds.
minSuccessful. Complete when the specified number of items have succeeded.
toleratedFailureCount. Complete when more than the specified number of failures have occurred.
toleratedFailurePercentage. Complete when the failure percentage exceeds the specified threshold (0.0 to 1.0).

When early termination is triggered, items that were never started have SKIPPED status with null for both result and error in the MapResult.

Each MapResultItem contains the following methods:

status: SUCCEEDED, FAILED, or SKIPPED.
result: the result value, or null if failed/skipped.
error: the error details as MapError, or null if succeeded/skipped.

Failed items store error details as MapError, a serializable record that survives checkpoint-and-replay cycles with the following methods that can be invoked:

errorType: Fully qualified exception class name (e.g., java.lang.RuntimeException).
errorMessage: The exception message.
stackTrace: Stack trace frames as a list of strings, or null.

One item’s failure does not prevent other items from completing. MapResult captures the failed items at their corresponding index.

Map operations are fully durable. On replay after interruption:

Completed items return cached results without re-execution.
Incomplete items resume from their last checkpoint.
Items that never started execute fresh.

Small results (< 256KB) are checkpointed directly, whereas large results are reconstructed from individual child context checkpoints on replay.

That’s it. There are no changes in the Infrastructure as Code part that we need to make. Now we can build and package our application with mvn clean package and deploy it with sam deploy. The deployment process can take up to 10 minutes because of the creation and mounting of S3 Files.

To test our Lambda durable function, we can navigate to the Lambda service, search for the AuthorContentExtractor or AsyncAuthorContentExtractor function, and go to the “Test” tab.

We need to pass the following sample JSON Event to it, which represents the author:

{
  "firstName": "Vadym",
  "lastName": "Kazulkin"
}

Then we can test it. After that, we go to the “Durable execution” tab and can see all the execution details:

Here we see that the operation with the name saveYouTubeVideoToPlayList-step and the subtype Map. We also see 2 iterations (for each of the 2 YouTube videos found) with the subtype MapIteration. Each iteration is executed as an individual durable step.

Let’s also look at event history:

Here we see all events (durable operations) in the right order required for the execution of our Lambda durable function.

There is also the mapAsync operation. It starts the map operation without blocking, returning a DurableFuture<MapResult>.

Conclusion

In this part of the series, we explored how to use the AWS Lambda Durable Execution SDK for Java to implement map and child context operations.

Still, our Lambda durable function itself contains too much business logic. Ideally, it should simply be the orchestrator and contain as little business logic as possible. That’s why, in the next part, we’ll move the logic for each step into a separate Lambda function. With that, we’ll explore how to invoke another Lambda function within the durable step. We’ll also show how to invoke multiple Lambda functions in parallel.

If you like my content, please follow me on GitHub and give my repositories a star!