Effective storage, retrieval and change management of application secrets

You may be storing application level secrets such as API keys or certificates as environment variables. As the environment variable are not hidden, this is a security vulnerability.

In this blog post, the author Timothy Jones presented a cost effective way to store and use application level secrets. You may want to go through the post to understand it in depth. I am summarizing the approach briefly as it may help us understand the alternative approach that I am presenting -

• Encrypt the secret parameters using KMS keys and store in Parameter Store as plain text. You can do this using an AWS CLI command.
• During deployment you are retrieving the parameters in their encrypted format. You can either user serverless plugin or through Cloud Formation Template.
• When you want to use hose parameters, you are decrypting their value and using them in your code.

This is a cost effective way to store and handle your secret variables. The major cost factor here is the decrypting of secrets using KMS which may cost you $3 if you are making 1000 requests per hour per month. The storage of parameters in Parameter Store is cheap. It does not cost anything to store information as you are going for Standard parameters.

The downside of the above approach is that if some of the parameters are changing quite often, you need to do frequent deployment so that your application can use updated parameter values.

I am presenting an approach using which you can manage effective secret storage, retrieval and seamless updating of parameters. The Parameter Store has a limit of maximum 100 requests per second. Although you can increase upto 1000 requests upon request, this throughput may not be enough for your application. To overcome this limitation, parameters will be cached in ElastiCache.

At high level, here is the approach

• Store the parameters as secret strings in Parameter Store
• When you need the parameters, check if the parameters are already available in cache. If not, fetch them from Parameter store and cache them in redis for subsequent usage
• Parameters will be stored in redis in decrypted form. You need to ensure that redis is encrypted at rest and in transit.

The code snippet below demonstrates how to make use of redis to store the parameter values.

redis-client.js

const redis = require('redis')
const { promisify } = require("util");

const client = redis.createClient({
  host: "master.****.****.***.cache.amazonaws.com",
  port: 6379,
  auth_pass: 'uQR********************************trQk',
  tls: {}
})

const getAsync = promisify(client.get).bind(client);
const setAsync = promisify(client.set).bind(client);
const delAsync = promisify(client.del).bind(client);

module.exports = {
  getAsync,
  setAsync,
  delAsync
}

Retrieval of parameters from parameter store —

const AWS = require('aws-sdk');
const ssm = new AWS.SSM({ region: 'us-east-1' });
const redisClient = require('./redis-client')

function getFromParameterStore(name) {
  return ssm.getParameter({
    Name: name,
    WithDecryption: true
  }).promise();
}

module.exports = async (name) => {
  const parameterValue = await getAsync(name);
  if (parameterValue) {
    return parameterValue;
  }
  try {
    const storedParameter = await getFromParameterStore(name);
    if (storedParameter) {
      const storedParameterValue = storedParameter.Parameter.Value;
      await setAsync(name, storedParameterValue);
    }
    return storedParameterValue;
  } catch (e) {
    console.error("Error when setting value in redis", e);
  }
}

In order to handle change of parameters, you can make use of AWS Event Bridge. You can configure handlers such as SNS topic, SQS queue or lambda to listen for the update events. For our scenario, a lambda function is appropriate which can clear redis.

Our goal is to remove a parameter value from redis when it is updated. It involves following steps —

• Create a lambda function to clear parameters from redis
• Add a rule in AWS Event Bridge to fire events when your parameters are updated.

The logic inside our lambda function is simple. For the parameter which is just updated, we are clearing it from redis. When the parameter is required inside your application, it is going to be retrieved from Parameter Store and will be cached in redis again.

const redisClient = require('./redis-client');
module.exports.handler = async event => {
  const parameterName = event.detail.name;
  await redisClient.delAsync(parameterName);
}

To add update events, go to AWS Event Bridge and create a new rule. Provide a name for the rule such as Update-Parameters.

You need to choose Event Pattern> Custom pattern. Then add the below pattern by editing the pattern —

{
  "source": ["aws.ssm"],
  "detail-type": ["Parameter Store Change"],
  "detail": {
    "name": [
      "/fcm/key"
    ],
    "operation": [
      "Update"
    ]
  }
}

You can include one or more parameter names under “detail.name” as a comma separated list. The operation “Update” denotes that you are interested in update events when the parameters are updated in Parameter Store.

Once you enter the event pattern, you need to configure the lambda you created in the first step as the event target. Choose Lambda function as the target for update events. Then select the lambda under “Functions” which is going to handle the events. Leave default values for other options.

When you save the changes, you are ready to go. The lambda will be invoked whenever parameters specified in the Update rule are changed.

Associated cost

If you are going for this approach, there is no additional cost if you are already using ElastiCache for your application. Otherwise, the smallest redis instance type t3.micro will cost around $37 per month.

To Conclude

If your parameters are rarely changing and you are fine with triggering a deployment for updating parameters, you can go for the option of storing encrypted parameters in Parameter Store and decrypting and using them at runtime. If you have frequent parameter changes and you want to avoid deployments, you go for this option which involves

• Storing parameters as secret text
• Caching parameter values in ElastiCache
• Using EventBridge to clear the cache using a lambda when parameters are updated.