How to run the Datomic transactor on Amazon ECS FARGATE

We have a continuous effort to reduce the complexity and maintenance cost of our application. Next on our road map was to get rid of explicitly managing a cluster of EC2 instances to provision docker images via ECS. Our goal was to run on ECS Fargate instead, a new launch type currently available in region us-east-1, which happens to be the region we are using. We have two images running on the cluster, the first one is our Clojure application. The other image is running the Datomic transactor. It took me a few days to figure out how to run the images on Fargate. Let me spare you valuable time by highlighting the configuration items that are important.

Service definition

The service definition is straightforward, the only notable thing here is the launch-type parameter, indicating Fargate

   :datomic-service (ecs/service {::ecs/service-name "datomic"
                                  ::ecs/task-definition (xref :datomic-task)
                                  ::ecs/cluster (xref :ecs-cluster)
                                  ::ecs/launch-type "FARGATE"
                                  ::ecs/network-configuration {::ecs/aws-vpc-configuration {::ecs/subnets [(xref :private1)]
                                                                                            ::ecs/security-groups [(xref :sg-private)]
                                                                                            ::ecs/assign-public-ip "DISABLED"} }
                                  ::ecs/desired-count 1})

Task definition

The task definition is a bit more tricky, things to note are transactor-role that is referenced twice (!), task-role-arn, execution-role-arn, family, network-mode and requires-compatibilities. Explanation in more detail below.

   :datomic-task (ecs/task-definition {::ecs/container-definitions [{::ecs/name "datomic"
                                                                     ::ecs/image "/datomic:0.9.5561.62"
                                                                     ::ecs/port-mappings [{::ecs/container-port 4334
                                                                                           ::ecs/host-port 4334}
                                                                                          {::ecs/container-port 4335
                                                                                           ::ecs/host-port 4335}
                                                                                          {::ecs/container-port 4336
                                                                                           ::ecs/host-port 4336}]
                                                                     ::ecs/environment [{:crucible.aws.ecs.task-definition.key-value-pair/name "licenseKey"
                                                                                         :crucible.aws.ecs.task-definition.key-value-pair/value ""}
                                                                                        {:crucible.aws.ecs.task-definition.key-value-pair/name "awsDynamodbTable"
                                                                                         :crucible.aws.ecs.task-definition.key-value-pair/value ""}
                                                                                        {:crucible.aws.ecs.task-definition.key-value-pair/name "awsDynamodbRegion"
                                                                                         :crucible.aws.ecs.task-definition.key-value-pair/value cf/region}
                                                                                        {:crucible.aws.ecs.task-definition.key-value-pair/name "awsTransactorRole"
                                                                                         :crucible.aws.ecs.task-definition.key-value-pair/value (xref :transactor-role)}
                                                                                        {:crucible.aws.ecs.task-definition.key-value-pair/name "awsPeerRole"
                                                                                         :crucible.aws.ecs.task-definition.key-value-pair/value (xref :peer-role)}
                                                                                        {:crucible.aws.ecs.task-definition.key-value-pair/name "memoryIndexThreshold"
                                                                                         :crucible.aws.ecs.task-definition.key-value-pair/value "32m"}
                                                                                        {:crucible.aws.ecs.task-definition.key-value-pair/name "memoryIndexMax"
                                                                                         :crucible.aws.ecs.task-definition.key-value-pair/value "512m"}
                                                                                        {:crucible.aws.ecs.task-definition.key-value-pair/name "objectCacheMax"
                                                                                         :crucible.aws.ecs.task-definition.key-value-pair/value "1g"}
                                                                                        {:crucible.aws.ecs.task-definition.key-value-pair/name "Xmx"
                                                                                         :crucible.aws.ecs.task-definition.key-value-pair/value "3g"}
                                                                                        {:crucible.aws.ecs.task-definition.key-value-pair/name "Xms"
                                                                                         :crucible.aws.ecs.task-definition.key-value-pair/value "3g"}]
                                                                     ::ecs/log-configuration {:crucible.aws.ecs.log-configuration/log-driver "awslogs"
                                                                                              :crucible.aws.ecs.log-configuration/options {"awslogs-group" "awslogs-ecs"
                                                                                                                                           "awslogs-region" "us-east-1"
                                                                                                                                           "awslogs-stream-prefix" "datomic"}}
                                                                     }]
                                            ::ecs/task-role-arn (xref :transactor-role) 
                                            ::ecs/execution-role-arn (xref :task-exec-role) 
                                            ::ecs/memory 3072
                                            ::ecs/cpu 1024
                                            ::ecs/family (join "-" [cf/stack-name "datomic"])
                                            ::ecs/network-mode "awsvpc"
                                            ::ecs/requires-compatibilities ["FARGATE"]})

task-role-arn / transactor-role

task-role-arn in the task definition must match the transactor-role in the transactor config file. This is what took me the longest to figure out.

The transactor-role when referenced in the task-role-arn property identifies which role is allowed to be assumed by containers in this task. This is prerequisite for datomic to be able to assume that role. It will not work without.

Note: I am still wondering whether or not datomic actually uses the value of this property. Maybe it is only used by the ensure-transactor scripts. If you know... send me a message.

execution-role-arn

We are hosting our docker image in an Amazon ECS repository. You need to give ECS access to that repository via a role. See the allowed ecr actions below. The other thing we do here, is allow access to cloudwatch so we can see the output of the task starting. This helps debugging:

   :task-exec-role (iam/role {::iam/assume-role-policy-document {::iam/version "2012-10-17"
                                                                      ::iam/statement [{::iam/effect "Allow"
                                                                                        ::iam/principal {::iam/service ["ecs-tasks.amazonaws.com"]}
                                                                                        ::iam/action ["sts:AssumeRole"]}]}
                                   ::iam/path "/"
                                   ::iam/policies [{::iam/policy-name "root"
                                                    ::iam/policy-document {::iam/version "2012-10-17"
                                                                           ::iam/statement [{::iam/action ["logs:CreateLogGroup"
                                                                                                           "logs:CreateLogStream"
                                                                                                           "logs:PutLogEvents"]
                                                                                             ::iam/effect "Allow"
                                                                                             ::iam/resource "*"}
                                                                                            {::iam/action ["ecr:*"]
                                                                                             ::iam/effect "Allow"
                                                                                             ::iam/resource "*"}]}}]})

family

This gives your task definition a reference when you want to create new versions of the task. We use this for continuous integration / continuous deployment from CircleCI. This is a snippet from our CircleCI workflow automatically creates new task definitions and cycles / updates the instances:

if [ "${CIRCLE_BRANCH}" == "develop" ]; then
  aws ecs list-task-definitions --family "application" --sort DESC > /tmp/task-definitions.json
  aws ecs describe-task-definition --task-definition `jq -r '.taskDefinitionArns[0]' /tmp/task-definitions.json` > /tmp/current-task-definition.json
  jq --arg TAG "${CIRCLE_SHA1}" '.taskDefinition | del(.taskDefinitionArn) | del(.revision) | del(.status) | del(.requiresAttributes) | del(.compatibilities) | .containerDefinitions[].image = "/test/next:" | .containerDefinitions[].image += $TAG' /tmp/current-task-definition.json > /tmp/task-definition-app.json
  cat /tmp/task-definition-app.json
  aws ecs register-task-definition --cli-input-json file:///tmp/task-definition-app.json > /tmp/new-task-definition.json
  aws ecs update-service --cluster "test" --service "application" --task-definition `jq -r '.taskDefinition.taskDefinitionArn' /tmp/new-task-definition.json`
fi

This is more relevant for the application image / instances and less relevant for the transactor image.

network-mode

aws-vpc is really the only possible value when you want to launch on Fargate

requires-compatibilities

Same thing here, this is a must.

Crucible

Wow, your cloudformation code looks like clojure! Yes indeed, we are using crucible written by brabster. Highly recommended. No more json, completion and lots of examples. If you need more of our cloudformation code, drop me a note.

Dockerfile

All of this resolves around a Dockerfile of course:

FROM clojure:lein-2.6.1-alpine

ENV DATOMIC_VERSION 0.9.5561.62
ENV DATOMIC_HOME /opt/datomic-pro-$DATOMIC_VERSION
ENV DATOMIC_DATA $DATOMIC_HOME/data

# .zip is not supported, please convert it to and commit a .tar.gz version
ADD datomic-pro-$DATOMIC_VERSION.tar.gz /opt
ADD config $DATOMIC_HOME/config
ADD bin $DATOMIC_HOME/bin

WORKDIR $DATOMIC_HOME

CMD ["bin/start.sh"]

VOLUME $DATOMIC_DATA

EXPOSE 4334 4335 4336

The start script is used to populate the datomic config file:

#!/bin/bash

printf "host=`ip -o -4 addr show eth1 | awk '{split($4,a,"/");print a[1]}'`n" >> config/base.properties
printf "license-key=$licenseKey n" >> config/base.properties

printf "aws-dynamodb-table=$awsDynamodbTablen" >> config/base.properties
printf "aws-dynamodb-region=$awsDynamodbRegionn" >> config/base.properties
printf "aws-transactor-role=$awsTransactorRolen" >> config/base.properties
printf "aws-peer-role=$awsPeerRolen" >> config/base.properties

printf "memory-index-threshold=$memoryIndexThresholdn" >> config/base.properties
printf "memory-index-max=$memoryIndexMaxn" >> config/base.properties
printf "object-cache-max=$objectCacheMaxn" >> config/base.properties

# Now we can run the transactor
bin/transactor -Xmx$Xmx -Xms$Xms config/base.properties

Here you finally see the binding between what we configure in the task definition and how this is passed on to Datomic.

Logging via cloudwatch

If you want all of the Datomic logging in cloudwatch, commit a logback.xml in the bin directory. It wil be added to the Docker image and used by Datomic. This is better than the log rotation configuration that is supported via the transactor-role:

      %d{yyyy-MM-dd HH:mm:ss.SSS} %-5level %-10contextName %logger{36} - %msg%n