Multimodal similarity search with Amazon Nova, S3 Vectors and Java

Amazon Nova 2 Multimodal Embeddings with Amazon S3 Vectors and AWS Java SDK

Amazon Nova 2 Multimodal Embeddings with Amazon S3 Vectors and AWS Java SDK – Part 1 Introduction

Amazon Nova 2 Multimodal Embeddings with Amazon S3 Vectors and AWS Java SDK – Part 2 Create and store text and image embeddings

Amazon Nova 2 Multimodal Embeddings with Amazon S3 Vectors and AWS Java SDK – Part 3 Create and store audio and video embeddings

Amazon Nova 2 Multimodal Embeddings with Amazon S3 Vectors and AWS Java SDK – Part 4 Implement similarity search

Introduction

In part 2, we covered creating text and image embeddings and in part 3, creating audio and video with Amazon Nova 2 Multimodal Embeddings and storing them in Amazon S3 Vectors using AWS Java SDK.

In this part of the series, we’ll take a look at the similarity search across all created multimodal embeddings in S3 Vectors. To those multimudal embeddings belong text, image, audio, and video embeddings.

You can find the code examples in my GitHub repository amazon-nova-2-multimodal-embeddings. Please give it a star if you like it, and follow me on GitHub for more examples.

Implement multimodal similarity search with Amazon Nova and Amazon S3 Vectors

We’ll reuse many parts of the process for creating and storing text and image embeddings described in part 2 and part 3. The relevant business logic of our sample application can still be found in the AmazonNovaMultimodalEmbeddings.

Here is how the relevant search method looks:

private static void search(String text, int topK) throws Exception {
    Float[] embeddings = createTextEmbeddings(text, "GENERIC_RETRIEVAL");
   
    var vd = VectorData.builder().float32(embeddings).build();
    var qvrRequest = QueryVectorsRequest.builder()
         .vectorBucketName(VECTOR_BUCKET).indexName(INDEX_NAME)
         .topK(topK)
	 .returnDistance(true).returnMetadata(true)
         .queryVector(vd).build();
    var qvResponse = S3_VECTORS_CLIENT.queryVectors(qvrRequest);

    for (var qov : qvResponse.vectors()) {
	    System.out.println("vector: " + qov);
    }
}

Let’s describe what’s happening here.

Create text embeddings for the search text

First of all, we invoke the createTextEmbeddings method described in part 2. We pass the search query as text and GENERIC_RETRIEVAL as the value of the embeddingPurpose. We set all other text embedding parameters exactly as we did it when we created embeddings (taskType as SINGLE_EMBEDDING and dimension as 384). For the complete embeddings request and response schema, I refer to the following article. Then we create text embeddings by invoking the corresponding Bedrock Model. For that, we need to build an InvokeModelRequest object by passing the text to create embeddings from. We also pass the JSON request converted to SDKBytes. Finally, we set the model id as amazon.nova-2-multimodal-embeddings-v1:0. Then, we use Bedrock Runtime Client to invoke the InvokeModelRequest synchronously and map the JSON response to the EmbeddingResponse and return its embeddings.

This all happens in the code part below:

Float[] embeddings = createTextEmbeddings(text, "GENERIC_RETRIEVAL");

Implement quering S3 Vectors

Then we build a VectorData object and set the returned embeddings from the previous step. After it, we build a QueryVectorsRequest object by passing the S3 bucket and index names, vector data as query vector. We also pass how many results to retrieve (topk). Finally we pass whether we’d like to return the distance and metadata, which we both set to true. Then we use the S3 Vectors Client to query vectors by passing the QueryVectorsRequest.

This all happens in the code part below:

var vd = VectorData.builder().float32(embeddings).build();
var qvrRequest = QueryVectorsRequest.builder()
    .vectorBucketName(VECTOR_BUCKET).indexName(INDEX_NAME)
    .topK(topK)
    .returnDistance(true).returnMetadata(true)
    .queryVector(vd).build();
var qvResponse = S3_VECTORS_CLIENT.queryVectors(qvrRequest);

The final part is to iterate over all vectors from the QueryVectorsResponse and print them:

var qvResponse = S3_VECTORS_CLIENT.queryVectors(qvrRequest);
for (var qov : qvResponse.vectors()) {
      System.out.println("vector: " + qov);
      }

Now let’s search by executing the following `search(“AWS Lambda”, 20)`. Here we search for the term “AWS Lambda” and would like to have the top 20 results. Remember that we created the following text embeddings: “AWS Lambda is a serverless compute service for running code without having to provision or manage servers. You pay only for the compute time you consume”. We also created AWS Lambda image embeddings in part 2, and audio and video embeddings from the AWS video about AWS Lambda function in part 3. We splitted both audio and video in 7 individual segments with 15 seconds each. For each of them, the separated embeddings were created.

Here is the output of our query:

vector: QueryOutputVector(Distance=0.36432654, 
Key=AWS-Lambda-explained-in-90-seconds-video_3, Metadata={})
vector: QueryOutputVector(Distance=0.40759963, 
Key=AWS-Lambda-explained-in-90-seconds-video_2, Metadata={})
vector: QueryOutputVector(Distance=0.4317584, 
Key=AWS-Lambda-explained-in-90-seconds-video_7, Metadata={})
vector: QueryOutputVector(Distance=0.44151443, 
Key=AWS-Lambda-explained-in-90-seconds-video_6, Metadata={})
vector: QueryOutputVector(Distance=0.4459787, 
Key=AWS-Lambda, Metadata={})
vector: QueryOutputVector(Distance=0.48943233, 
Key=AWS Lambda Definition, Metadata={})
vector: QueryOutputVector(Distance=0.5942307, 
Key=Azure Functions Definition, Metadata={})
vector: QueryOutputVector(Distance=0.663301, 
Key=Azure-Functions, Metadata={})
vector: QueryOutputVector(Distance=0.6920378, 
Key=AWS-Lambda-explained-in-90-seconds-video_5, Metadata={})
vector: QueryOutputVector(Distance=0.710474, 
Key=AWS-Lambda-explained-in-90-seconds-video_4, Metadata={})
vector: QueryOutputVector(Distance=0.73109514, 
Key=AWS-Lambda-explained-in-90-seconds-video_1, Metadata={})
vector: QueryOutputVector(Distance=0.9187513, 
Key=Life Definition, Metadata={})
vector: QueryOutputVector(Distance=1.0514107, 
Key=AWS-Lambda-explained-in-90-seconds-audio_6, Metadata={})
vector: QueryOutputVector(Distance=1.0748283, 
Key=AWS-Lambda-explained-in-90-seconds-audio_7, Metadata={})
vector: QueryOutputVector(Distance=1.0772036, 
Key=AWS-Lambda-explained-in-90-seconds-audio_4, Metadata={})
vector: QueryOutputVector(Distance=1.0832013, 
Key=AWS-Lambda-explained-in-90-seconds-audio_5, Metadata={})
vector: QueryOutputVector(Distance=1.093506, 
Key=AWS-Lambda-explained-in-90-seconds-audio_1, Metadata={})
vector: QueryOutputVector(Distance=1.1019071, 
Key=AWS-Lambda-explained-in-90-seconds-audio_3, Metadata={})
vector: QueryOutputVector(Distance=1.1073054, 
Key=AWS-Lambda-explained-in-90-seconds-audio_2, Metadata={})

Interpret search results

The impression is kind of a mixed one. We see that the results are sorted by the distance, and we see the key also printed out. There is no printed metadata, as we didn’t set it when storing the embeddings in the S3 Vectors. We see that certain video embeddings (15-second segments) are at the top (having a lower distance) as the corresponding Lambda image (the key is “AWS-Lambda”) and the text (the key is “AWS Lambda Definition”). But I also see that all audio segments are completely at the bottom of the search results. For them I’d expect to have a much lower distance (similar to their corresponding video segments). And also the Azure Function embeddings (the key is “Azure Functions Definition” for the text embeddings and the key is “Azure-Functions” for the image embeddings) have quite low distance, which I’d expect to be higher. Of course, the explanation for this can be that Azure Functions is similar to AWS Lambda – a Serverless service with similar benefits and features, but offered by a different cloud provider (Microsoft Azure). But should have gotten an impression of how the similarity search works.

Conclusion

In this part of the series, we covered the implementation of the similarity search across all (text, image, audio, and video) created embeddings in S3 Vectors.

If you like my content, please follow me on GitHub and give my repositories a star!