Cloudformation KeyValuePair List as a parameter
Multi tool use
Cloudformation KeyValuePair List as a parameter
When creating ECS infrastructure we describe our Task Definitions with CloudFormation. We want to be able to dynamically pass environment variables as a parameter to the template. According to the docs, Environment has a KeyValuePair type, but CloudFormation parameters do not have this type.
We can not hardcode Environment variables to the template, because this template is used as a nested stack so environment variables will be dynamically passed inside it.
The only possible way I see so far is to pass all arguments as a CommaDelimitedList, and then somehow parse and map it using CloudFormation functions. I can Fn::Split every entity in key and value, but how to dynamically build an array of KeyValuePair in CloudFormation?
Or maybe there is an easier way, and I'm missing something? Thanks in advance for any ideas.
@RodrigoM, amount of key-value pairs are dynamic, but in our case, it is up to 20 environment variables. Those environment variables are being passed to Docker container if this is what you are asking for.
– Ivan
Dec 15 '17 at 2:54
I was just wondering where in the process are they referenced. Are the values referred to in Docker file or in the run time application? both?
– Rodrigo M
Dec 15 '17 at 14:32
@RodrigoM In runtime application only, ECS is working with already baked dockerfiles and pass variables to containers
– Ivan
Dec 16 '17 at 19:07
You could write a script that generates the CloudFormation template based on your required environment variables. That script can then use AWS API to also deploy the stack.
– kichik
Dec 17 '17 at 18:30
2 Answers
2
You may want to consider using the EC2 Parameter Store to create secured key/value pairs, which is supported in CloudFormation, and can be integrated with ECS environments.
AWS Systems Manager Parameter Store
AWS Systems Manager Parameter Store provides secure, hierarchical
storage for configuration data management and secrets management. You
can store data such as passwords, database strings, and license codes
as parameter values. You can store values as plain text or encrypted
data. You can then reference values by using the unique name that you
specified when you created the parameter. Highly scalable, available,
and durable, Parameter Store is backed by the AWS Cloud. Parameter
Store is offered at no additional charge.
While Parameter Store has great security features for storing application secrets, it can also be used to store nonsensitive application strings such as public keys, environment settings, license codes, etc.
And it is supported directly by CloudFormation, allowing you to easily capture, store and manage application configuration strings which can be accessed by ECS.
This template allows you provide the Parameter store key values at stack creation time via the console or CLI:
Description: Simple SSM parameter example
Parameters:
pSMTPServer:
Description: SMTP Server URL eg [email-smtp.us-east-1.amazonaws.com]:587
Type: String
NoEcho: false
SMTPServer:
Type: AWS::SSM::Parameter
Properties:
Name: my-smtp-server
Type: String
Value: !Ref pSMTPServer
Any AWS runtime environment (EC2, ECS, Lambda) can easily securely retrieve the values. From the console side, there is great Parameter manager interface that maintains parameter version history. Its intergated with IAM, so permissions are controlled with standard IAM policy syntax:
{
"Action": [
"ssm:GetParameterHistory",
"ssm:GetParameter",
"ssm:GetParameters",
"ssm:GetParametersByPath"
],
"Resource": [
"arn:aws:ssm:us-west-2:555513456471:parameter/smtp-server"
],
"Effect": "Allow"
},
{
"Action": [
"kms:Decrypt"
],
"Resource": [
"arn:aws:kms:us-west-2:555513456471:key/36235f94-19b5-4649-84e0-978f52242aa0a"
],
"Effect": "Allow"
}
Finally, this blog article shows a technique to read the permissions into a Dockerfile at runtime. They suggest a secure way to handle environment variables in Docker with AWS Parameter Store. For reference, I am including their Dockerfile here:
FROM grafana/grafana:master
RUN curl -L -o /bin/aws-env https://github.com/Droplr/aws-env/raw/master/bin/aws-env-linux-amd64 &&
chmod +x /bin/aws-env
ENTRYPOINT ["/bin/bash", "-c", "eval $(/bin/aws-env) && /run.sh"]
With that invocation, each of the parameters are available as an environment variable in the container. You app may or may not need a wrapper to read the parameters from the environment variables.
It does not really solve the problem of inserting a list of parameters into the template, does it? I need not only to pass env variable values, but keys as well
– Ivan
Dec 16 '17 at 19:06
Well yes you can include 20 parameters in a template of course. Is that what you mean? What type of keys? Its a general parameter store, so it can store anything. We use them to store ssh keys, among other types of data
– Rodrigo M
Dec 17 '17 at 1:45
They are equal in function to the TaskDefinition ContainerDefinition KeyValue pairs you mentioned needing in CloudFormation. SSM paramters are in CloudFormation today. You can also refer to the values in other stacks by using Fn::GetAtt(parameter,'value'). Seems like it should fit your given requirements no?
– Rodrigo M
Dec 17 '17 at 1:54
KeyValuePair has 'Key' and 'Value' properties. I need to insert not only 'Value' but also 'Key'. And amount KeyValuePairs is not known when you write a template, they all dynamic. So the whole array of KeyValuePairs must be inserted. Is that explains my problem? Maybe I was not clear enough in the initial questions.
– Ivan
Dec 17 '17 at 19:18
To be fair I think @rodrigo-m is right in that using external config service like parameter store allows you to lookup required config values as needed. If you don't want to build the lookup into your container you could use a sidecar container to do that and link to your primary container.
– fortuna
yesterday
I was facing the same problem ,I needed to create a lambda resource with environment variables.
We decided to fix initial set of environment variable and keys name are also decided in advance.
So I added four parameters , and used Ref for values while keeping fixed keys name.
By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.
What client/service consumes the key pair from the environment? How many keypairs need to be injected as parameters?
– Rodrigo M
Dec 14 '17 at 21:14