Bitbucket Pipelines: Custom Docker Image with Amazon Elastic Container Service

Bitbucket Pipelines accelerate .NET Builds

Introduction

Bitbucket Pipelines is an excellent CI/CD tool that allows users of Bitbucket to automate the Continuous Integration and Continuous Delivery of their software, including:

  • Building code
  • Running automated tests
  • Auditing code for vulnerabilities
  • Performing static code analysis
  • Packaging code artifacts
  • Publishing code
  • Deploying code

Challenge

One of the challenges developers may face after adopting Bitbucket Pipelines for multiple repositories is the need to add steps to their pipelines that install software not included in their chosen base Docker image. Examples include installing:

Over time, developers may end up with the same steps copied & pasted between pipelines and repositories, which can become burdensome to manage, and add to the runtime of pipelines.

Solution

One way to address this challenge is by creating a custom Docker image to use for pipelines. These images can be hosted on Docker Hub or, in the case of this example, Amazon Elastic Container Service.

In this post, we will demonstrate:

  • Creating a custom Docker image for building .NET Core code
  • Using Bitbucket Pipelines to tag and push images to Amazon ECS
  • Using the custom Docker image in Bitbucket Pipelines

Creating the Docker Image

The .NET 6.0 Docker Image provided by Microsoft does not including common packages such as Node.js and NPM, or utilities for compression. These dependencies are useful for publishing web applications. In addition to these dependencies, we also use the following tools in many of our builds at IDMWORKS:

Let’s take a look at what a Dockerfile would look like that includes the above tools.

FROM mcr.microsoft.com/dotnet/sdk:6.0

USER root

# Install zip, unzip, JRE, and Node.js
RUN curl -sL “https://deb.nodesource.com/setup_16.x” | bash –
RUN apt-get update \
    && apt-get install –yes –no-install-recommends \
        zip \
        unzip \
        openjdk-11-jre \
        nodejs

# Install the Amazon AWS CLI
RUN curl “https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip” -o “awscliv2.zip”
RUN unzip awscliv2.zip \
    && ./aws/install \
    && rm -r ./aws/

# Install the Library Manager CLI and Sonar Scanner
RUN dotnet tool install -g Microsoft.Web.LibraryManager.Cli \
    && dotnet tool install –global dotnet-sonarscanner

# Clean up temp files
RUN rm -r /tmp/*

# Remove unneeded packages
RUN apt-get -qy autoremove \
    && rm -rf /var/lib/apt/lists/*

We won’t go into too much detail here as it’s beyond the scope of this post; this is a fairly straight-forward Docker file that extends the image mcr.microsoft.com/dotnet/sdk:6.0 by installing our common build tools.

Pushing the Docker Image

To begin using the new Docker image, the first thing we’ll want to do is a create a new repository (either private or public) in Amazon Elastic Container Registry. In this example, we’ll create a repository for the fictional company Hooli named hooli/dotnet-6-sdk.

Once we’ve created the repository, we can click the View push commands button in the Amazon AWS Management console to see the commands we can use to push images from the command line, e.g.:

aws ecr get-login-password –region us-east-1 | docker login –username AWS –password-stdin 1234567890.dkr.ecr.us-east-1.amazonaws.com
docker build -t hooli/dotnet-6-sdk .
docker tag hooli/dotnet-6-sdk:latest 1234567890.dkr.ecr.us-east-1.amazonaws.com/hooli/dotnet-6-sdk:latest
docker push 1234567890.dkr.ecr.us-east-1.amazonaws.com/hooli/dotnet-6-sdk:latest

Next, we’ll introduce a bitbucket-pipelines.yml file so that we can use conventional Git workflows and CI/CD to automate tagging and pushing the image to Amazon ECS.

image:
  name: amazon/aws-cli:latest

initContainerRegistry: &initContainerRegistry
  aws –version &&
  aws configure –profile ecr set aws_access_key_id $AWS_HOOLI_ACCESS_KEY_ID &&
  aws configure –profile ecr set aws_secret_access_key $AWS_HOOLI_SECRET_ACCESS_KEY &&
  aws ecr get-login-password –region us-east-1 –profile ecr | docker login –username AWS –password-stdin 1234567890.dkr.ecr.us-east-1.amazonaws.com

buildTagAndPush: &buildTagAndPush
  docker build -t hooli/dotnet-6-sdk:$img_version . &&
  docker tag hooli/dotnet-6-sdk:$img_version 1234567890.dkr.ecr.us-east-1.amazonaws.com/hooli/dotnet-6-sdk:$img_version &&
  docker push 1234567890.dkr.ecr.us-east-1.amazonaws.com/hooli/dotnet-6-sdk:$img_version

pushLtsImage: &pushLtsImage
  step:
    name: Push image tagged “lts”
    caches:
      – docker
    script:
      – *initContainerRegistry
      – img_version=”lts”
      – *buildTagAndPush
    services:
      – docker

pushLatestImage: &pushLatestImage
  step:
    name: Push image tagged “latest”
    caches:
      – docker
    script:
      – *initContainerRegistry
      – img_version=”latest”
      – *buildTagAndPush
    services:
      – docker

pushVersionedImage: &pushVersionedImage
  step:
    name: Push image tagged with version
    caches:
      – docker
    script:
      – *initContainerRegistry
      – img_version=$BITBUCKET_TAG
      – *buildTagAndPush
    services:
      – docker

pipelines:
  branches:
    master:
      – <<: *pushLtsImage
    develop:
      – <<: *pushLatestImage
  tags:
    ‘*.*.*’:
      – <<: *pushVersionedImage

As before, we’ll need to provide values for the following pipeline repository variables:

  • AWS_HOOLI_ACCESS_KEY_ID: access key ID for a service account in the Hooli AWS account
  • AWS_HOOLI_SECRET_ACCESS_KEY (secured): secret access key for a service account in the Hooli AWS account

Conclusion

With these pieces put together, we now have a custom Docker image, hosted in Amazon ECS, that can be used to build .NET Core code without having to install additional dependencies in each pipeline. In addition, the Docker image itself has a set of pipelines that automate the Continuous Integration and Continuous Delivery of the image, automatically tagging and pushing the image using conventional Git workflows.

In future posts, we’ll look at related topics surrounding the automation of CI/CD with Bitbucket Pipelines and .NET Core code, such as:

Author: Nathanial Woolls, IDMWORKS, IAM Architect 

Of note, we’re using the amazon/aws-cli image so that the AWS CLI is available, and we’re using the docker Bitbucket Pipelines service so we have access to Docker commands. We’re also using YAML anchors to introduce re-usable pipeline sections.

To make use of this bitbucket-pipelines.yml file, we’ll need to enable Bitbucket Pipelines for the repostory, and provide values for the following pipeline repository variables:

  • AWS_HOOLI_ACCESS_KEY_ID: access key ID for a service account in the Hooli AWS account
  • AWS_HOOLI_SECRET_ACCESS_KEY (secured): secret access key for a service account in the Hooli AWS account

With this configuration in place:

  • Pushing (or merging) code to the develop branch will build & push a Docker image to AWS ECR with the tag “latest”.
  • Pushing (or merging) code to the master branch will build & push a Docker image to AWS ECR with the tag “lts”.
  • Pushing a Git tag with a version — such as 1.0.0 — will build & push a Docker image to AWS ECR with a tag based on the Git tag.

Using the Docker Image

Once one of the above Git workflows has been performed in order to push the tagged image to Amazon ECS, new and existing pipelines for building .NET Core code can be updated with the following at the top of their bitbucket-pipelines.yml file:

image:
  name: 1234567890.dkr.ecr.us-east-1.amazonaws.com/hooli/dotnet-6-sdk:latest
  aws:
    access-key: $AWS_HOOLI_ACCESS_KEY_ID
    secret-key: $AWS_HOOLI_SECRET_ACCESS_KEY

As before, we’ll need to provide values for the following pipeline repository variables:

  • AWS_HOOLI_ACCESS_KEY_ID: access key ID for a service account in the Hooli AWS account
  • AWS_HOOLI_SECRET_ACCESS_KEY (secured): secret access key for a service account in the Hooli AWS account

Conclusion

With these pieces put together, we now have a custom Docker image, hosted in Amazon ECS, that can be used to build .NET Core code without having to install additional dependencies in each pipeline. In addition, the Docker image itself has a set of pipelines that automate the Continuous Integration and Continuous Delivery of the image, automatically tagging and pushing the image using conventional Git workflows.

In future posts, we’ll look at related topics surrounding the automation of CI/CD with Bitbucket Pipelines and .NET Core code, such as:

Author: Nathanial Woolls, IDMWORKS, IAM Architect