Available Docker Images

This repository contains Dockerfiles for Docker images used in both software projects and research projects (such as PHRs). The containers are available in the repository’s associated container registry.

The Dockerfile of some of the more commonly used images are shown and described below. Note that the ARG values displayed are defaults. Different values may be used in the CI build as defined in .gitlab-ci.yml.

Docker builder image

The docker image is used for building other docker images. This image is built from the bash image, since many of scripts used in our CI processes require bash constructs.

ARG bashvers=latest
FROM bash:${bashvers}

ARG dockervers
RUN apk add --no-cache docker=${dockervers}

Ubuntu

This image with the ubuntu OS serves as the base image for many other images. It includes docker and AWS cli tools.

ARG ubuntuVersion=latest
FROM ubuntu:$ubuntuVersion

RUN apt-get update && \
    apt-get install -y \
    apt-transport-https \
    ca-certificates \
    curl \
    unzip \
    gnupg-agent \
    software-properties-common \
    file

# Install Docker
RUN curl -fsSL https://download.docker.com/linux/debian/gpg | apt-key add -
RUN add-apt-repository "deb [arch=amd64] https://download.docker.com/linux/ubuntu bionic stable"
RUN apt-get install docker-ce -y
RUN apt-key fingerprint 0EBFCD88

# Install AWS CLI
RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install
RUN which aws
RUN /usr/local/bin/aws --version

Antora

This image contains the documentation site generation Antora and the following extensions:

The antora image is based on a node image, so it has npm and other utilities as well.

ARG ANTORA_VERS

FROM antora/antora:${ANTORA_VERS}

RUN yarn global add @antora/lunr-extension asciidoctor-kroki
# RUN npm install --global asciidoctor asciidoctor-kroki

Nix images

An image for building and developing using NixOS.

ARG nixVersion=
FROM nixos/nix:$nixVersion

COPY docker/nix/nix.conf /etc/nix/nix.conf
COPY docker/nix/upload-to-cache.sh /etc/nix/upload-to-cache.sh
ARG NIX_CACHE_PRIVATE_KEY
RUN echo "$NIX_CACHE_PRIVATE_KEY" > /etc/nix/key.private

Nix caches

Nix supports remote caches (store) for sharing build artifacts.

NoviSci nix store

We serve a nix store on S3 in the following bucket:

nsstatdev-main-usae1-nix-cache

The public key for this cache is:

nsstatdev-main-usae1-nix-cache-1:3qt3qqlXhyl2HGK8UE1Eh12NEmoyK8mx81uDWDAKPn4=

Our nix docker image is configured to upload to this cache after each build, using a post build hook as described here.

The upload_to_cache.sh script expects that the NIX_CACHE_PRIVATE_KEY environment variable be set when the container is run. This should be set for each GitLab repository where this image is run in CI/CD Settings → Variables (at the group level).
The IAM role statdev-nix-cache-ci in the research dev account has the appropriate AWS permissions to get/put from our S3 cache. AWS credential variables should be set for each GitLab repository where this image is run in CI/CD Settings → Variables.

Other caches

The configuration file (nix.conf) copied into the docker image includes our internal cache as one of several substituters (sources of pre-built artifacts). Additional substituters include the default nix cache, dhall caches, and iohk caches recommended for haskell.nix.

Test container locallly

Build and run this container locally with:

docker build \
  --build-arg nixVersion=2.10.3 \
  --file docker/nix/Dockerfile \
  --tag foo .
docker run -it \
  -e NIX_CACHE_PRIVATE_KEY=<PUTNIXKEYHERE> \
  -e AWS_ACCESS_KEY_ID=<KEY> \
  -e AWS_SECRET_ACCESS_KEY=<PUTAWSKEYHERE> \
  -e AWS_DEFAULT_REGION=us-east-1 \
  foo

Example stuff to do in the container to be sure things are working:

nix build nixpkgs#hello

Dhall

This image contains the Dhall interpreter and various Dhall executables such as dhall-to-json, dhall-to-yaml, and dhall-to-csv. It is based from the ubuntu image, and thus has all of its utilities as well.

FROM registry.gitlab.com/targetrwe/epistats/nsstat/nsbuild/ubuntu:latest

ARG DHALL_VERS
ARG DHALL_BASH_VERS
ARG DHALL_JSON_VERS
ARG DHALL_CSV_VERS

RUN mkdir install-dhall \
    && cd install-dhall \
    && curl -L "https://github.com/dhall-lang/dhall-haskell/releases/download/$DHALL_VERS/dhall-$DHALL_VERS-x86_64-linux.tar.bz2" | \
    tar --extract --bzip2 \
    && ./bin/dhall --help
RUN cd install-dhall \
    && cp ./bin/dhall /usr/local/bin

RUN cd install-dhall \
    && curl -L "https://github.com/dhall-lang/dhall-haskell/releases/download/$DHALL_VERS/dhall-bash-$DHALL_BASH_VERS-x86_64-linux.tar.bz2" | \
    tar --extract --bzip2 \
    && ./bin/dhall-to-bash --help install-dhall \
    && cp ./bin/dhall-to-bash /usr/local/bin 

RUN cd install-dhall \
    && curl -L "https://github.com/dhall-lang/dhall-haskell/releases/download/$DHALL_VERS/dhall-json-$DHALL_JSON_VERS-x86_64-linux.tar.bz2" | \
    tar --extract --bzip2 \
    && ./bin/dhall-to-json --help install-dhall \
    && cp ./bin/dhall-to-json /usr/local/bin \
    && cp ./bin/dhall-to-yaml /usr/local/bin

RUN cd install-dhall \
    && curl -L "https://github.com/dhall-lang/dhall-haskell/releases/download/$DHALL_VERS/dhall-csv-$DHALL_CSV_VERS-x86_64-linux.tar.bz2" | \
    tar --extract --bzip2 \
    && ./bin/dhall-to-csv --help install-dhall \
    && cp ./bin/dhall-to-csv /usr/local/bin;

Haskell images

Development Haskell

An image for building and developing Haskell projects. It includes AWS cli tools.

ARG GHCVERS=
FROM haskell:${GHCVERS}
RUN cabal update
RUN cabal install hspec-discover
RUN cabal install hlint
RUN cabal install ShellCheck
RUN cabal install doctest
# See https://github.com/haskell/stylish-haskell/issues/405
# for reason for constraint
RUN cabal install --constraint "stylish-haskell +ghc-lib" stylish-haskell
RUN cabal install shelltestrunner-1.9

RUN apt-get update && \
    apt-get install -y \
    unzip 

RUN curl "https://awscli.amazonaws.com/awscli-exe-linux-x86_64.zip" -o "awscliv2.zip"
RUN unzip awscliv2.zip
RUN ./aws/install
RUN which aws
RUN /usr/local/bin/aws --version

Static Haskell

An image for building statically-linked Haskell executables.

ARG GHCVERS=8107
ARG V=v22
FROM utdemir/ghc-musl:${V}-ghc${GHCVERS}

RUN cabal update
RUN cabal install cmdargs aeson hspec text witherable --lib
RUN cabal install hspec-discover

R images

Image for building and developing R packages and projects.

r-fixedvers

This is a "batteries-included" docker image with R, the tidyverse, many of NoviSci’s internal packages, and some cli tools like awscli.

# The main image for R development, based from rocker/verse

ARG RVERS
FROM rocker/verse:${RVERS}

# The `DEBIAN_FRONTEND=noninteractive` is added to silence a complaint during
# the `awscli` install (see e.g. https://stackoverflow.com/a/56569081/5518304).
#
# * The AWS CLI (via `awscli`) is used by many of the R package CI scripts
# * `bzip2` is needed for the Dhall installation script
# * `devscripts` includes `checkbashisms` which is needed during `R CMD check`
#   if you include a configure script for any of your packages.
# * `libssh2-1` and `libgit2-dev` are needed for git2r
# * `pandoc` is used to build package vignettes
RUN apt-get update
RUN DEBIAN_FRONTEND=noninteractive apt-get install -y \
    awscli \
    bzip2 \
    devscripts \
    libssh2-1 \
    libgit2-dev \
    pandoc

# Install the Dhall family of applications
COPY docker/r/install-dhall.sh /
RUN bash /install-dhall.sh

# Note that we need to redeclare `RVERS` because we are in a different stage
# than for the first declaration
ARG RVERS
COPY docker/r/r-fixedvers-installs.R r-fixedvers-installs.R
RUN Rscript r-fixedvers-installs.R

# Install NoviSci packages.
# The BUST_CACHE argument is used to bust the docker cache.
# In gitlab-ci.dhall the associated job is passed a gitlab CI variable
# to bust the cache.
# The cache needs to be busted because we want
# latest versions of our packages installed.
# Since the docker build scheduled to run daily,
# we should have a
ARG BUST_CACHE
RUN echo ${BUST_CACHE}

COPY docker/r/r-fixedvers-novisci-installs.R r-fixedvers-novisci-installs.R
RUN Rscript r-fixedvers-novisci-installs.R
CI jobs building r-fixedvers-based docker image will be run on a daily schedule, and the CI timestamp is used to bust the docker cache. The does mean the installations of novisci packages happen even if they don’t need to be (i.e. the versions haven’t changed). The advantage of this approach is that it’s simple, and the R docker build jobs can fit in the mold of docker-build-push/job.dhall.

r-devel

This package is meant to be used for testing packages for CRAN release. CRAN policies required that R CMD check --as-cran be run on the current version of R when submitted to CRAN. You can do that with this image.

# A docker image for running R CMD check --as-cran on a current version
# of r-devel to comply with CRAN policies when submitted to CRAN.

FROM rocker/r-devel:latest

# add pandoc (necessary for building vignettes)
RUN apt update && apt-get install pandoc -y

COPY docker/r/r-devel-installs.R install.R
RUN Rscript install.R

Rust images

An image for building and developing Rust projects. It includes AWS cli tools.

ARG rustVersion
FROM rust:${rustVersion}

RUN apt-get update 
RUN apt-get install -y awscli libssh2-1
RUN cargo install cargo-audit

Linting and formatting

This image contains the following linters and formatters for various languages.

There is currently no AsciiDoc linter, but it may be worth considering adding one at another time. There is some support for AsciiDoc in e.g. Vale through Asciidoctor. Also see this Linting Markdown And Documentation blog post for general information regarding linting Markdown and other literate programming formats.
# Note that it is important to fix the Ubuntu version (rather than using
# `latest`) because the R installation is version-specific
FROM ubuntu:20.04

# This ARG is used solely as a mechanism for busting the image cache and
# rebuilding all of the layers in the image
ARG LINT_YYYYMMDD

# Avoid user interaction with `apt` (through `dpkg`): https://askubuntu.com/a/1013396/447630
ENV DEBIAN_FRONTEND=noninteractive

RUN apt update
# Install `add-apt-repository`: https://computingforgeeks.com/how-to-install-add-apt-repository-on-debian-ubuntu/
RUN apt install -y software-properties-common dirmngr apt-transport-https lsb-release ca-certificates
# Install other common utilities
RUN apt install -y wget curl


# Haskell ----------------------------------------------------------------------
# Includes hlint, brittany, ShellCheck, hadolint

# The following PPA is the recommended place for Ubuntu packages: https://downloads.haskell.org/~debian/
# GHC installation instructions: https://launchpad.net/~hvr/+archive/ubuntu/ghc
RUN add-apt-repository -y ppa:hvr/ghc && apt update
RUN apt install -y ghc-8.10.4 cabal-install-3.4
ENV PATH="/opt/ghc/bin:/opt/cabal/bin:$PATH"

# This appears to the be recommended way to install hlint, brittany, and
# hadolint. There is Debian package available for ShellCheck, but since such
# packages tend to not be current we install it here as well
RUN cabal update && cabal install hlint brittany hadolint ShellCheck

# Add configuration files
COPY docker/lint-polyglot/config-files/.hlint.yaml /
RUN mv /.hlint.yaml "$HOME"


# R ----------------------------------------------------------------------------
# Includes styler, lintr

# R installation instructions:
# * R 4.1: https://cran.r-project.org/bin/linux/ubuntu/fullREADME.html
# * R 3.6: https://cran.r-project.org/bin/linux/ubuntu/olderreleasesREADME.html

# This link provides the packages for the current major version of R version 4.
# Packages for older versions of R are not provided for Ubuntu 20 or later
ARG SOURCES='deb https://cloud.r-project.org/bin/linux/ubuntu focal-cran40/'
RUN echo "$SOURCES" >> '/etc/apt/sources.list'
RUN wget -qO- https://cloud.r-project.org/bin/linux/ubuntu/marutter_pubkey.asc > /etc/apt/trusted.gpg.d/cran_ubuntu_key.asc && apt update
RUN apt install -y r-base r-base-dev

# PPA instructions: https://launchpad.net/~c2d4u.team/+archive/ubuntu/c2d4u4.0+
RUN add-apt-repository -y ppa:c2d4u.team/c2d4u4.0+ && apt update
RUN apt install -y r-cran-styler

# For some reason lintr is not available in the PPA, so manually download some
# of the dependencies before installing. In particular, the lintr install will
# fail unless the following dependencies (more accurately, the underlying system
# packages) are satisfied prior to installation. The other packages are included
# solely to speed the build.
# * curl
# * xml2
# * openssl
RUN apt install -y r-cran-curl r-cran-xml2 r-cran-openssl r-cran-stringr r-cran-sys r-cran-lazyeval r-cran-mime
RUN Rscript -e 'install.packages("lintr")'


# Dhall ------------------------------------------------------------------------
# Includes dhall

# Expected to be passed in through `--build-args`
ARG DHALL_VERS

RUN mkdir install-dhall \
    && cd install-dhall \
    && curl -L "https://github.com/dhall-lang/dhall-haskell/releases/download/$DHALL_VERS/dhall-$DHALL_VERS-x86_64-linux.tar.bz2" | \
    tar --extract --bzip2 \
    && ./bin/dhall --help
RUN cp install-dhall/bin/dhall /usr/local/bin && rm -rf install-dhall


# Rust -------------------------------------------------------------------------
# Includes clippy

# Rust installation instructions: https://forge.rust-lang.org/infra/other-installation-methods.html#other-ways-to-install-rustup
RUN curl https://sh.rustup.rs -sSf | sh -s -- -y
ENV PATH="$HOME/.cargo/bin:$PATH"


# Node JS ----------------------------------------------------------------------
# Includes markdownlint
# Use `markdown-it` to run markdownlint

# Node JS installation instructions:
# * https://nodejs.org/en/download/package-manager/#debian-and-ubuntu-based-linux-distributions
# * https://github.com/nodesource/distributions/blob/master/README.md#debinstall
RUN curl -fsSL https://deb.nodesource.com/setup_17.x | bash -
RUN apt install -y nodejs

# For some reason this command needs to be run in the /usr/app directory:
# https://stackoverflow.com/q/57534295.
WORKDIR /usr/app
RUN npm install markdownlint --save-dev
ENV PATH="$(npm bin):$PATH"
WORKDIR /


# Python -----------------------------------------------------------------------
# Includes yamllint

# Install pip. Note that this assumes that we already have Python 3 installed
RUN wget https://bootstrap.pypa.io/get-pip.py
RUN apt install -y python3-distutils
RUN python3 get-pip.py && rm get-pip.py

# yamllint installation instructions: https://github.com/adrienverge/yamllint
RUN python3 -m pip install yamllint