Secrets Manager vs Parameter Store

AWS gives you two ways to store application configuration: Secrets Manager and Systems Manager Parameter Store. Both use KMS (Key Management Service) to encrypt the data. Both can store arbitrary configuration data. Both use IAM (Identity and Access Management) policies to control access. So which should you pick?

There isn’t one “right” answer; which you choose depends to a large degree on how you manage your applications. This post looks at some of the considerations for choosing between them. But first, a short description of how they work.

Secrets Manager lets you store a single string or binary value of up to 64kbytes, giving it a name. The entire string is encrypted using KMS, with either a default or customer-specified KMS key. Typically, the string is a JSON object, and the AWS Console will parse the string and allow you to view or edit it as individual name-value pairs. If you access the secret via the CLI or in your program you’ll need to parse it yourself.

Parameter Store (like many people, I omit the “Systems Manager” part of its name) stores individual values using a hierarchical key. You can have keys like /database/username and /database/password, and either retrieve them individually or retrieve all keys that start with /database. The values can be simple strings, comma-separated lists (which you have to parse), or encrypted strings (which also support default and custom KMS keys). When you retrieve the data, you can choose whether or not to decrypt encrypted values.

Permissions

Secrets are all or nothing: either you have permission to decode the secret or you don’t. Sometimes this is justified: for example, if your secret contains an SSH private key. But with other secrets, such as database connection information, you might want to block access to the password but not other information such as the hostname (as I know from bitter experience, having once spent a holiday weekend wondering why a production app wasn’t working properly, only to learn later that it was configured to point to a test database).

Parameter Store, with its hierarchical naming convention, lets you apply different permissions based on the individual parameter name. For example, to allow users to view all of the /database parameters except /database/password, you can use the following IAM policy:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Sid": "BaseParameterPermissions",
            "Effect": "Allow",
            "Action": [
                "ssm:DescribeParameters",
                "ssm:GetParametersByPath",
                "ssm:GetParameters",
                "ssm:GetParameter"
            ],
            "Resource": "arn:aws:ssm:*:123456789012:parameter/example/database/*"
        },
        {
            "Sid": "ProtectSensitiveParams",
            "Effect": "Deny",
            "Action": "*",
            "Resource": "arn:aws:ssm:*:123456789012:parameter/example/database/password"
        }
    ]
}

Advantage: Parameter Store

Secrets Rotation

The underlying idea of secrets rotation is that you should regularly change the credentials you use to access other services, to limit the “blast radius” if they are compromised. For more information about why this is a good thing, listen to this SERadio podcast.

Secrets Manager provides this “out of the box,” using a Lambda function. For supported AWS database engines, it provides a pre-built Lambda; for unsupported databases or other services, you have to write the Lambda yourself. While it’s nice to have a pre-written Lambda function, there’s nothing that prevents you from writing your own function that updates credentials managed by Parameter Store, and invoking it via a CloudWatch scheduled event.

That said, my personal preference for database credentials uses neither approach: instead, it uses IAM to generate a password that only lasts 15 minutes. Unfortunately, this feature is only available for a limited number of RDS and Aurora database versions.

Advantage: Secrets Manager (slight)

Creation with CloudFormation

CloudFormation lets you create Parameter Store parameters, as long as they don’t contain secure strings.

For Secrets Manager it lets you create the secret and generate a value to store in it. For example to create the “master login” for an RDS database:

MasterUser:
  Type:                               "AWS::SecretsManager::Secret"
  Properties:
    Name:                             !Sub "${ServerName}-MasterUser"
    Description:                      !Sub "RDS master username and password"
    GenerateSecretString:
      SecretStringTemplate:           |
                                      {
                                        "username": "postgres",
                                        "database": "postgres"
                                      }
      GenerateStringKey:              "password"
      ExcludePunctuation:             true
      PasswordLength:                 64

Here I use a hardcoded username and default database name, and Secrets Manager generates the password. It will be 64-characters long, and consist of alphanumeric characters only (which makes it easier to select in the Console on those — hopefully rare — occasions when you need to connect directly to the database).

If you create the database instance in the same stack, you can refer to this secret without its value ever being exposed. You can also use a SecretTargetAttachment resource that will update the secret with additional information, such as the hostname of the database instance.

Advantage: Secrets Manager

Consumption by CloudFormation

Once you’ve created secrets and/or parameters, you’ll probably want to use them in your CloudFormation scripts. There are two ways to do this: SSM parameter types and dynamic references.

An SSM parameter type leverages the existing CloudFormation parameter mechanism, with a twist: when you set the parameter you provide a Parameter Store key rather than the actual value. When you reference the parameter, CloudFormation retrieves the value using that key. In my opinion, this is most useful when the actual value is not easy to recognize. For example, an ACM certificate doesn’t have a human-friendly ARN, so referencing it with the key /prod/cert makes it easy to see that you’re correctly configured.

ACMCertificate:
  Description:                        "A Parameter Store key where the actual certificate ARN can be found"
  Type:                               "AWS::SSM::Parameter::Value<String>"

By comparison, dynamic references support both Secrets Manager and Parameter Store, and are applied directly in the resource definition. For example, using the master database user secret from the last section, here are the relevant references in an RDS resource:

MasterUsername:                   !Sub "{{resolve:secretsmanager:${MasterUser}:SecretString:username}}"
MasterUserPassword:               !Sub "{{resolve:secretsmanager:${MasterUser}:SecretString:password}}"
DBName:                           !Sub "{{resolve:secretsmanager:${MasterUser}:SecretString:database}}"

With Secrets Manager, referencing a secret value is straightforward: you specify the ARN of the secret (here via substitution) and the field within that secret. Parameter Store references are somewhat more complex, in that they have to explicitly identify the parameter version.

Parameter Store references are also significantly more limited in where they an be used: a plain string parameter can be used anywhere, but a secure string can only be used for password fields in specific resources. By comparison, you can reference a Secrets Manager secret anywhere, even in places where the secret may be exposed, such as the environment variables used by a Lambda.

Advantage: Secrets Manager (but beware the possibility of leaking secrets).

Consumption by Other Services

The “Twelve-Factor App” recommends that all application configuration happens through environment variables. However, that leaves open the question of how you set those variables. While it’s convenient to configure variables as part of deployment, using CloudFormation or Terraform, that can expose sensitive values. The typical alternative, however, is to programmatically retrieve the values when the application starts, which can be annoying during development (when you usually want to point at locally-hosted services).

The Elastic Container Service (ECS) shows an alternative approach: you can configure your ECS task definitions with references to either Secrets Manager or Parameter Store. The values are retrieved by the service when the task starts, and are provided to your code as environment variables. They are never visible outside of the application code itself.

This feature still has some rough edges: for example, whether a Secrets Manager secret is parsed into name value pairs depends on whether you’re running on Fargate or EC2, and which version of the ECS container agent you’re using. But the idea is sound, the rough edges should be smoothed out over time, and hopefully the same idea will be implemented in Lambda and other services.

Advantage: Parameter Store (for now)

Cost

A lot of people seem to be concerned about the cost of Secrets Manager; one person I know claimed that it was “insane at scale,” and there are questions on Stack Overflow that ask for the best way to store secrets “other than secrets manager because the cost is too high.” So, let’s compare the costs:

	Monthly Cost	Cost per Request
Secrets Manager	$0.40	$5 per 1,000,000
“Standard” Parameters	free	free (standard throughput) $5 per 1,000,000 (higher throughput)
“Advanced” parameters	$0.05	$5 per 1,000,000 (standard throughput) $10 per 1,000,000 (higher throughput)

This isn’t an apples-to-oranges comparison. For one thing, a single secret can hold the same information as dozens of parameters (64k vs 4k for standard parameters and 8k for advanced). You can also have an unlimited number of secrets, while you’re limited to 10,000 “standard tier” parameters and 100,000 “advanced tier.”

Since the per-request cost is actually higher for Parameter Store, I suppose the per-secret cost is the what concerns most people. That’s not helped by the AWS pricing example that creates 5,000,000 secrets that only live for an hour (I consider that particular example to be one of using the wrong tool for the job). In a more normal use-case, where you store the secrets that your application needs to operate, you’ll find that Secrets Manager will be a tiny fraction of your AWS spend.

Advantage: neither

Wrapping Up: Keep Your Secrets Secret!

That sounds like useless advice, but it’s surprisingly easy to leak a secret. For example, Terraform lets you retrieve a secret or secure string, but then stores that value in the “tfstate,” which is stored as plain text by default. Understanding your tools is critical to avoiding leaks.

On the other hand, some things that look like they expose secrets can be configured so that they don’t. For example, I earlier mentioned that you could use a dynamic reference to assign a secret value to a Lambda environment variable. If you do nothing else, that value will be visible to anyone that can retrieve the Lambda configuration. However, Lambda gives you the option to encrypt the environment using an explicit KMS key. If a user doesn’t have permission for that key, they can see the Lambda configuration but not the environment values.

And remember: whether you choose Secrets Manager or Parameter Store, you’re better off than if you used a file with secrets stored in plain sight.