Set up CloudWatch Events Rules that sends SNS Notification on Step Status Change for EMR Clusters with a Specific Name

When we try to implement Cloudwatch notifications via SNS/email that are triggered when a specific EMR cluster has step state changes, the event pattern structure works when used with clusterId but fails when used with ClusterARN/cluster name. There is no documented approach that would specify ARN or name of a cluster in the Event Pattern used while creating CloudWatch Events Rule.

As a workaround to achieve this use case, you may consider creating a lambda function that gets triggered whenever a new cluster is created. The Lambda function can then create new CloudWatch Events Rule corresponding to that cluster if it has the same "name" as expected. This event rule can specify an Event pattern with the Cluster-id of the relevant clusters (retrieved from the event that triggered the lambda) with the target as an SNS topic.

Following are the steps that can be followed to implement this workaround

1. Create an SNS topic that sends notifications.

2. Create a lambda function using the script attached.

3. Specify the following Environment Variables to the lambda function

   • CLUSTER_NAME: Name of the Cluster (A prefix from the cluster name may also work)
   • SNS_ARN: ARN of the SNS topic created in the first step.

4. Create a CloudWatch Event Rule with the following Event Pattern and set the Lambda function created in step 2 as its target. This CloudWatch event will trigger the lambda function whenever a new Cluster is created.

   // Event Pattern: { "detail-type": [ "EMR Cluster State Change" ], "source": [ "aws.emr" ], "detail": { "state": [ "STARTING" ] } }

   // Lambda Script: 'use strict';

   const AWS = require('aws-sdk'); const uuidV4 = require('uuid/v4');

   exports.handler = (event, context, callback) => { const eventName = event.detail.name; let response = {};

      if (eventName.search(process.env.CLUSTER_NAME) === -1) {
          response = {
              statusCode: 400,
              body: JSON.stringify('Cluster name not found'),
          };
          callback(null, response);
          return;
      }
   
      let cloudwatchevents = new AWS.CloudWatchEvents();
      let params = {
          Name: uuidV4(),
          EventPattern: '{"source": ["aws.emr"],"detail-type": ["EMR Step Status Change"],"detail": {"clusterId": ["' + event.detail.clusterId +'"]}}',
          State: 'ENABLED',
      };
   
      let putRulePromise = cloudwatchevents.putRule(params).promise();
      putRulePromise.then((data) => {
          let params = {
              Rule: data.RuleArn.split("/")[1],
              Targets: [
                  {
                      Arn: process.env.SNS_ARN,
                      Id: uuidV4(),
                  },
              ]
          };
   
          return cloudwatchevents.putTargets(params).promise();
      }).then((data) => {
          console.log("Hello " +  data);
          response = {
              statusCode: 200,
              body: JSON.stringify('Done!'),
          };
      }).catch((err) => {
          console.log(err);
          response = {
              statusCode: 200,
              body: JSON.stringify('Failed'),
          }
      });
   
      callback(null, response);

   };