Communication between two cloud services is not new concept but in one of the requirement I face some challenges to find way for upload file from cosmos database to S3 Bucket because there is couple of documentation and tutorials available for AWS to Azure data transfer but very less for vice versa.

After some tutorials and study I found one of the easiest way to perform this task using Serverless Azure Functions and triggers. the image below will be the simple architecture of the solution.

If you want to do same solution then just follow below steps:

Step 1 : Create new Azure Function

Refere this link for step by step implementation.

Step 2 : Add custom logic

You need to modify or add logic for fetch documents from cosmos database as per your requirement. CosmsDBTrigger function work same as program in C# or any language so you can use same classes and namespaces which you use in normal application like CosmosDBClient, QueryDefination etc. please reffer sample code given below in this article.

Step 3 : Add code for connect S3 Bucket

Once you finish with main logic for fetch data from cosmosdb you need to connect S3 bucket where you want to pass that data. so you need below credentials with you,

• S3SecretKey
• S3AccessKey
• S3BucketName
• s3RequestKey -> this will be file name which you want to be given.

Using all this details you can connect to S3 bucket and send data to S3 bucket also you will get response from aws service about success/failure of operation.

Step 4 : Network Setup

If AWS service have private network implemented then it will not allow to accept request from outside like azure portal, so you need to get necessary port whitelisted.

Step 5: Publish code on Azure

If you write/create function in visual studio/vs code then you need to publish that code on portal. kindly refer below links for publish azure function on portal with multiple ways. docs.microsoft.com/en-us/azure/azure-functi..

Code Sample

[FunctionName("SendDataToS3")]
public static async RunAsync(CosmosDBTrigger(
    databaseName: "<Your Cosmos DB Name>",
    collectionName: "<Collection Name>",
    ConnectionStringSetting = "CosmosDBString", // dont mention connection string here keep it as it is    
    LeaseCollectionName = "leases",
    CreateLeaseCollectionIfNotExists = true,
    FeedPollDelay=5)] IReadOnlyList<Document> input,
    ILogger log)
{
    try
   {
        //this function will run automatically everytime when data will be change/update in cosmos db
        if (input != null && input.Count > 0)
        {
             log.LogInformation("Documents Modified " + input.Count);
        }

        //implement you database operation related logic here like read data from cosmos db etc
        string query = "<Write a query which fetch data> ";
        QueryDefination queryDef = new QueryDefination(query);

        // create CosmosDBClient
        var client = new CosmosDBClient(Environment.GetEnvironmentVariable("CosmosConnectionString"));
        var container = client.getContainer("<Your database id>", "<your container id>");

        using (FeedIterator<string> iterator = container.GetItermQueryIterator<string>(queryDef))
        {
            while (iterator.HasMoreResults)
            {
                 list.add(iterator.ReadNextAsync());
            }
        }

        //then connect to S3 bucket with following code
        var s3Secret = Environment.GetEnvironmentVariable("S3Secret");
        var s3AccessKey = Environment.GetEnvironmentVariable("S3AccessKey");
        var s3BucketName = Environment.GetEnvironmentVariable("S3BucketName");
        var s3RequestKey = "Give key name (Any)"; // if  you are trying to upload file then give any name for that file.

        // Create AmazonS3Client using secrets
        var awsClient = new AmazonS3Client(s3AccessKey, s3Secret, RegionEndPoint.USEast1);

        var request = new PutObjectRequest()
        {
             Key = s3RequestKey;
             BucketName = s3BucketName;
             ContentBody = JsonConvert.SerializeObject(cosmosDbData);
        }
        PutObjectResponse responseFromAWS = await awsClient.PutObjectAsync(request);

        // check status of upload operation 
        if (awsResponse.HttpStatusCode.Equals(HttpStatusCode.OK))
        {
             log.LogInformation("Scenarios file uploaded successfully");
        }
        else
        {
            log.LogError("Failed to upload file on S3 Bucket");
        }
   }
   catch (Exception e)
    {
        log.LogError(e);
        throw;
    }
}

Package Reference

1. AWSSDK.S3
2. Microsoft.Azure.Cosmos
3. Microsoft.NET.Sdk.Functions

Addtional Links

• c-sharpcorner.com/article/how-to-create-and..

Best Practices:

This task contains sensitive data like cosmos database secrets, aws secrets so we need to save all this secret in key-vault, kindly refer below links to configure key-vaults with azure functions.

daniel-krzyczkowski.github.io/Integrate-Key.. azure.microsoft.com/en-us/blog/simplifying-..

Thank you :)

Ajay Samgir