Dynamic Configuration Management using Amazon S3 Events and AWS AppConfig

AWS AppConfig gives you the ability to create, manage, and deploy configuration changes separate from code. This helps you to avoid deploying the service repeatedly for small changes.

AppConfig Configuration Profile supports Amazon S3 as one of the configuration stores and allows validation of the configurations before deploying them to the service. This allows us to build a robust system that addresses use cases in which we need to offload the configuration update to some external entity.

This post demonstrates a workflow that lets you dynamically update the configurations stored in Amazon S3. Here, I assume that you have a general familiarity with the following AWS Services:

• Amazon S3
• AWS Lambda
• AWS AppConfig

The aforementioned diagram outlines the architecture of this workflow:

1. A user uploads the configuration file to S3.
2. When the file upload completes, a Lambda function is triggered. This function fetches the latest version of the S3 file and then starts deployment for the AppConfig Application.
3. Before the deployment of the AppConfig application starts, another Lambda function gets triggered. This lambda function parses the data from the configuration file and then validates it based on some predefined schema rules.
   • If the file in invalid, the Application deployment is cancelled and it fetches the previous valid version of the configuration file when the Client (EC2 instance in the diagram) invokes the GetConfiguration API.
   • If the file is valid, the deployment starts, and when it completes, it fetches the latest version.

Triggering Lambda function on Amazon S3 File Upload

Amazon S3 lets you publish notifications when certain events happen in your bucket such as 'new object created events' and 'object deleted events'. We can leverage this functionality for our use case to trigger a lambda function, whenever the configuration file is uploaded.

1. Create a Lambda function that invokes the StartDeployment API of AppConfig with the latest version of the S3 object.
2. Create an S3 Bucket with Bucket Versioning enabled.
3. From the Bucket Properties, add an event notification with:
   • Events: [PUT]
   • Prefix: the S3 object prefix
   • Suffix: the file name, feature-file.yml
     • This ensures that only the update for feature-file.yml file triggers the lambda function
   • Send to: Lambda Function
   • Lambda: Select the function from Drop-down, DeployAppConfig

Lambda function

import json
import boto3
import os

appconfig = boto3.client('appconfig')

def lambda_handler(event, context):
    versionId = event['Records'][0]['s3']['object']['versionId']
    applicationId = os.environ['APPLICATION_ID']
    configurationProfileId = os.environ['CONFIGURATION_PROFILE_ID']
    deploymentStrategyId = os.environ['DEPLOYMENT_STRATEGY_ID']
    environmentId = os.environ['ENVIRONMENT_ID']
    try:
        response = appconfig.start_deployment(
            ApplicationId=applicationId,
            EnvironmentId=environmentId,
            DeploymentStrategyId=deploymentStrategyId,
            ConfigurationProfileId=configurationProfileId,
            ConfigurationVersion=versionId,
        )
    except Exception as e:
        print('Error starting deployment of AppConfig Application with ' + 
        'ApplicationId: {} EnvironmentId: {} '.format(applicationId, environmentId) + 
        'DeploymentStrategyId: {} ConfigurationProfileId: {} '.format(deploymentStrategyId, configurationProfileId) +
        'ConfigurationVersion: {}'.format(versionId))
        raise e

To ensure that the configuration data store in the S3 bucket is semantically and syntactically valid, you can create a Configuration Profile Validator as an AWS Lambda function.

Whenever you start the AppConfig deployment, the specified validator is triggered that validates the input file. If the data is invalid, it stops the deployment, giving clients the last valid configuration. This ensures that the configurations are always valid and provides us with a robust mechanism.

Summary

AWS AppConfig is a powerful tool that is used thousands of times a day inside of Amazon and AWS and has now been externalized for their customers to use. In this post, we have demonstrated one simple yet effective use case of this service.

If you have feedback about this post, submit comments in the Comments section below.