Getting Started

Antora

The purpose of this section is to provide guidance on compiling a 'local version' of your Antora project. Instructions and a link to an example test script are provided.

Instructions

  1. Install Node.js (unless it’s already on your system) by following the instructions at Intalling Node.js via a package manager.

  2. Install Antora (unless it’s already installed) by following the instructions at Install Antora.

  3. Compile a 'local version' of your Antora project.

    1. Some projects will have a test_antora.sh script in the scripts directory.

    2. If no test_antora.sh script exists, you can create one by copying the test_antora.sh file into the project.

    3. Open a terminal in the project root directory.

    4. Run ./scripts/test_antora.sh.

    5. Copy the file path reported in the output and paste it into the web browser of your choice to view the generated documents.

Compiling a 'local version' of your Antora project.

When writing non-WYSIWYG documents, it is often helpful to regularly compile the target format of the documents so that content, layout, and formatting can be regularly reviewed. One of the key features of the Antora documentation system is the ability to consume content from multiple sources, however this also has the effect of making it more difficult to align everything needed to compile the document in toto. Thus, when working on a particular component belonging to a larger Antora project we have found it convenient to create a local version of the project containing just the current project’s component. The Antora scaffolding surrounding the content of the local version is "placeholder code" that doesn’t get used in the larger Antora project.

Amazon Web Services (AWS)

The purpose of this section is to provide guidance on AWS setup. Instructions on AWS Console Login are provided.

AWS Console Login

Following these instructions will allow you to access AWS services through the web console.

  • Request access to AWS by emailing ithelpdesk@targetrwe.com. You will need access to the console as well as an Identity and Access Management (IAM) user account.

  • Once access is granted, go to the Target single-sign on portal.

  • Select "AWS" from the dropdown and click "Sign In".

  • Add your credentials and click "Sign In".

  • In the AWS services block, select the desired service (EC2, S3, etc.) from either "Recently visited" or "View all services" or by using the search bar.

Requesting AWS services through the web API

AWS web API overview

AWS provides a web API to access its services. For example, one common task is to download a file from AWS S3 onto the main stat machine. Typically we don’t utilize the web API directly, but instead use a library or application that provides a more convenient interface for the API. The AWS API has bindings in many programming languages, including the following:

  • See the AWS Command Line Interface web page for a set of AWS command-line tools.

  • See the list of per-language libraries provided by AWS at the Tools to Build on AWS web page.

  • See the amazonka GitHub page for an unofficial Haskell AWS library.

  • See the PAWS web page for an unofficial R AWS library.

Authentication through the web API

The AWS authentication scheme is documented in the Signing and authenticating REST requests page, however applications and libraries that provide access to the AWS web API are free to implement a user interface for specifying the authentication information in whatever manner they see fit. That being said, the AWS CLI application provides a system for providing authentication credentials that is often emulated by other applications and libraries and consequently defines a loose de facto standard. Thus we will provide instructions to get you set up using the AWS CLI authentication system, however you will need to check whether a given AWS library or application that you are using supports your particular authentication setup.

Creating an IAM user token

To access AWS services through the web API from your local machine you will need to create an Identity and Access Management (IAM) access key. If you don’t care about accessing the AWS web API from your local machine then this step isn’t strictly necessary as there is another approach for providing credentials on the main stat machine as described in the Providing credentials for the AWS CLI (main stats server) section.

To create an IAM access key this you can follow the "To create an access key" instructions in the Managing access keys (console) section of the Managing access keys for IAM users page. Note that you will have to have been provided access to an IAM user account, so if you don’t see your name in the IAM Users section as described in the previous link then you will have to email ithelpdesk@targetrwe.com to request access.

  1. Log in to the AWS web console by navigating to the Target RWE SSO page and clicking through until you reach the AWS console splash page.

  2. Once you make it to the AWS console splash page, navigate to the IAM services page either by clicking on "Services" on the top ribbon and then subsequently clicking on "All Services" and then scrolling to and clicking on "IAM", or alternatively by typing "IAM" into the search bar on the top ribbon and clicking on "IAM".

  3. Once you make it to the IAM page click "Users" on the left-hand sidebar.

  4. Once your browser has updated the page to show a list of users then click on your name from among the list. Note that in the resulting IAM user page that you may see some "You need permissions" messages — you can safely ignore these messages.

  5. Once you have loaded your IAM user page then click on the "Security Credentials" tab.

  6. Click on the "Create New Access Key" button and choose the option to download a CSV file which will have the form described below.

The CSV file provided by the AWS IAM service will have the following form.

Access key ID,Secret access key
EZJ46Z2K3IPAAGKDFWIQ,rCUcmQu6AlG8Tuou1o+AxTYHQWa3I/NHr+OL+Nhp
Providing credentials for the AWS CLI (local machine)

The full specification for the AWS CLI authentication system is provided in the AWS CLI Configuration Variables page. There are many ways to provide your credentials, but we suggest constructing an AWS configuration file. This file should be located at ~/.aws/config (%UserProfile%\.aws\config or ~\.aws\config in Windows Command shell or PowerShell, respectively) and should have the following form. The values for aws_access_key_id and aws_secret_access_key are obtained from the file you downloaded as part of the Creating an IAM user token section.

[default]
aws_access_key_id = <Access key ID>
aws_secret_access_key = <Secret access key>
Providing credentials for the AWS CLI (main stats server)

You can provide the same credentials for the main stats server that you use for your local machine, however the main stats server is endowed with its own AWS credentials so as an alternative you can use those credentials instead of your own. The stats server’s credentials may have more permissions than your user account so that may be a good default choice, however we occasionally run into situations where libraries or applications don’t authenticate properly with these credentials (the R cloudyr tools are one such example), in which case you might need to use your user credentials instead. If you want to use the main stats server’s credentials then your ~/.aws/config file would look like the following.

[default]
credential_source = Ec2InstanceMetadata
Installing the AWS CLI (local machine)

To install the AWS CLI follow the instructions provided in the Installing or updating the latest version of the AWS CLI page. Note that the AWS CLI is already installed on the main stats server.

Using the AWS CLI

Some AWS CLI documentation is available at the What is the AWS Command Line Interface? page. In particular note the Using Amazon S3 with the AWS CLI and Using Amazon EC2 with the AWS CLI subsections.

The complete API reference is provided through the application help documentation. For example, you can use the commands aws help, aws s3 help, or aws s3 cp help to get help for any top-level AWS CLI command, for the AWS CLI s3 subcommand, or for the AWS CLI s3/cp sub-subcommand, respectively.

To test whether your setup is working you can try a command like aws s3 cp s3://docs.novisci.com/concepts/d03f731c863a0da0a6f3b04166f3659a172ccad3.yaml ADHD.yaml and see if that saves a file ADHD.yaml to your current working directory.

Dhall

Dhall application installation

The following instructions are provided as an example to install several core Dhall applications on a macOS system. See the Dhall releases page for the appropriate URL to download a different application or version (and make the appropriate modifications to the instructions). Other approaches to installing Dhall applications are available and are described in more detail in the Dhall application installation extended discussion.

In total, the following instructions will install the dhall, dhall-to-json, dhall-to-yaml, and dhall-lsp-server applications.

  1. Open up a terminal

  2. Download the Dhall application by running wget https://github.com/dhall-lang/dhall-haskell/releases/download/1.41.1/dhall-1.41.1-x86_64-macos.tar.bz2.

  3. Decompress and extract the application by running tar -xvf dhall-1.41.1-x86_64-macos.tar.bz2.

  4. Copy the extracted file to a location on your shell’s exectuble path by running sudo cp bin/dhall /usr/local/bin.

  5. Repeat steps 2-4 to install the dhall-to-json and dhall-to-yaml, and dhall-lsp-server applications (note that the first download contains two applications) for the following files:

    • https://github.com/dhall-lang/dhall-haskell/releases/download/1.41.1/dhall-json-1.7.10-x86_64-macos.tar.bz2

    • https://github.com/dhall-lang/dhall-haskell/releases/download/1.41.1/dhall-lsp-server-1.0.18-x86_64-macos.tar.bz2

Setting up editor support for the Dhall language

Editor-specific setup instructions for the Dhall language are available below. Regardless of which editor you are using, be sure to additionally install the Dhall LSP server as described in the Dhall application installation section.

Setting up editor support for the Dhall language for VSCode

Instructions for setting up Dhall language support for VSCode can be found in the VSCode section in the Dhall text editor configuration guide as well as in the vscode-dhall-lsp-server GitHub page.

Setting up editor support for the Dhall language for Vim

Instructions for setting up Dhall language support for Vim can be found in the Vim section in the Dhall text editor configuration guide.

  • Note that this section is based on the use of vim-plug so if you are a user of another LSP server like coc.nvim, ALE, or neovim’s built-in server then will have to adapt the instructions to conform to your setup.

Setting up editor support for the Dhall language for Emacs

Instructions for setting up Dhall language support for Emacs can be found in the Emacs section in the Dhall text editor configuration guide.

  • Note that this section is based on the use of lsp-mode so if you are an eglot user then you will have to adapt the instructions to conform to your setup.

  • Some distributions such as Spacemacs and Doom Emacs have an available Dhall layer/module, so if you use such a distribution you may consider starting with the configuration that is provided to you by the distribution and modifying it from there.

GitLab

The purpose of this section is to provide guidance on GitLab setup.

Instructions

  • Ensure that you can log in the to GitLab console through a web browser at the NoviSci GitLab website. If you are unable to log in then you may need to email ithelpdesk@targetrwe.com to have them add you as a user.

  • Ensure that you can access the repositories that you will be working with. If you try to access a particular repository and get an error message like HTTP Basic: Access denied then ask to be granted permission to access the nsResearch and nsStat groups by sending a request to the #techsupport Slack channel.

  • Optionally set up an SSH key to save yourself from having to enter a username and password each time you perform an action (taken from https://docs.gitlab.com/ee/user/ssh.html).

    • Open a terminal in your home directory.

    • Verify the SSH version is greater than or equal to 6.5 by running ssh -V`.

    • Generate an SSH key.

      • In the terminal, run ssh-keygen -t ed25519 -C <your_email_address@targetrwe.com>.

      • Click enter.

      • Specify a passphrase.

    • Adding the ssh key to GitLab (detailed instructions here)

      • Copy the contents of the .ssh file created in the step above.

      • In GitLab, go to Preferences > SSH Keys.

      • Paste the contents of the .ssh file in the "Key" box.

      • Add a description in the "Title" box.

      • Click "Add Key"

    • Verify the connection.

      • In the terminal, run ssh -T git@gitlab.example.com.

      • If an "are you sure?" message pops up,

        • Type "yes" and click "Enter".

        • Rerun ssh -T git@gitlab.example.com.

      • You should receive a welcome message.

      • If no messsage is received, troubleshoot by running ssh -Tvvv git@gitlab.example.com.

  • In your IDE, clone the desired repo.

Haskell Setup

The purpose of this section is to provide guidance for setting up Haskell. Minimal, step-by-step instructions are provided in the Quick Start section. Troubleshooting guidance may be found in the Troubleshooting section. Guidance on IDE setup may be found in the IDE-Specific Guidance section.

Quick Start

The purpose of this section is to provide generic, step-by-step instructions for installing Haskell. These instructions are project agnostic. Reference the project developer guide and/or user guide for specific instructions.

Three applications will be installed: GHCup, GHC and Cabal. GHCup is a utility for installing applications in the Haskell toolchain. GHC is a Haskell compiler. Cabal is a system for building and packaging Haskell code.

If you are using a Windows system then you may be queried at various times during or after the installation process as to whether the system’s virus protection application should allow a given application to run. Please select "Allow" for any such queries. Also see Command not found/recognized error message (Windows specific) for troubleshooting problems related to virus protection systems.

If you receive error messages at any point during setup, please consult the Troubleshooting section.

Instructions

  1. Open a terminal and point to your home directory.

  2. GHCup installation

    1. In your terminal, run the command provided to you at the GHCup landing page.

    2. If using Windows OS, when prompted with Where to install to (this should be a short Path, preferably a drive like 'C:\') input the appropriate path (using the suggested default will likely be fine).

    3. If using Windows OS, when prompted with Specify Cabal directory (this is where haskell packages end up) input the appropriate path (using the suggested default will likely be fine).

    4. If using a Unix-like system, when prompted with Do you want ghcup to automatically add the required PATH variable to your shell startup file? select "Yes, prepend".

    5. When prompted with Do you want to install haskell-language-server (HLS)? select "Yes".

    6. When prompted with Do you want to install stack? see the specific setup instructions for your project(s). If the projects you will be working on use Cabal, then Stack is not needed (though there is no harm in installing it).

    7. If using Windows OS, when prompted with Do you want GHCup to install a default MSys2 toolchain? select "Yes" unless you already MSys2 installed.

    8. After installation finishes, restart your terminal.

    9. Check that GHCup installed correctly by typing ghcup --help. If the command is not recognized, check the Troubleshooting section first. If there are still issues, request help in the techsupport Slack channel.

Installing individual toolchain components

You will have likely installed each of the following toolchain components as part of the GHCup installation. However, you may at some point want to install a different version of one of those components. If that is the case then the following are the instructions for installing those components. It is worth noting that even if you have a particular version of a given component installed then there is no harm in asking GHCup to install it again because it will be smart enough to essentially make it a no-operation the second time around.

  1. GHC installation

    1. The version to install is project-specific, but likely 8.10.7. To determine the version of GHC to install for your project consult the cabal.project file (for Cabal projects), or if it’s not specified there then it could be also be mentioned in the .gitlab-ci.yml file in the project repository. If you are working on multiple projects, then you may need to install multiple versions of GHC.

    2. Once the appropriate version has been determined, type ghcup install ghc <VERSION> in the terminal and press Enter.

    3. Verify that ghc has been installed by typing ghc-<VERSION> --help. If the command is not recognized, check the Troubleshooting section first. If there are still issues, contact a member of the stats dev team for help.

  2. Cabal installation

    1. Type ghcup install cabal 3.6.2.0 in the terminal and press Enter.

    2. Verify that cabal has been installed by typing cabal-3.6.2.0 --help. If the command is not recognized, check the Troubleshooting section first. If there are still issues, contact a member of the stats dev team for help.

  3. HLS installation. Usually you will want to install whatever the recommended version of HLS is according to ghcup list.

    1. Type ghc install hls in your terminal and press Enter.

Troubleshooting

LLVM error message (macOS specific)

If you receive the error message below:

<no location info>: error:
    Warning: Couldn't figure out LLVM version!
             Make sure you have installed LLVM between [9 and 13)

then you need to install a different build of LLVM.

If you have Homebrew installed, you can run the code below

brew install llvm

If you are not a Homebrew user, check the LLVM website for alternative installation instructions.

You will also want to add the following line, amended to include the appropriate path, to your shell startup file: ${HOME}/.bashrc, ${HOME}/.zshrc, etc.

export PATH="/opt/homebrew/opt/llvm/bin:$PATH"

After adding the above line (potentially with the appropriate modifications) restart your shell. Also see Ensuring applications are on the search path for more specific instructions regarding regarding shell search paths.

Also see Help wanted for LLVM config for Haskell on Mac on Reddit for some additional discussion.

Command not found/recognized error message (Windows specific)

If you attempt to run <APPLICATION NAME> --help, where the APPLICATION NAME is ghc, cabal, ghci, etc., and you get an error message like the following, it is possible that you are running the command in MinGW or WSL. Please try the command in PowerShell instead.

bash: ghc: command not found

Alternatively, suppose you attempt to run <APPLICATION NAME> --help in PowerShell and get an error message like the following.

ghc : The term 'ghc' is not recognized as the name of a cmdlet, function, script file, or operable program.
Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

First, ensure that you have restarted PowerShell since you performed the installation. Next, check the bin folder in your GHCup installation directory; this will often default to C:\ghcup\bin. This folder should contain files like <APPLICATION NAME>, and <APPLICATION NAME>.shim.

  1. If the GHCup bin folder doesn’t exist or doesn’t contain a collection of files like the ones described above, then there was likely a problem with the application installation and you may want to wipe everything clean and retry the installation.

  2. Next, check that GHCup bin folder is on your search path. Enter $Env:Path into PowerShell and ensure that you see the appropriate GHCup bin folder in the output.

  3. If the GHCup bin folder does exist and is on the search path, then you may need to whitelist the application you are trying to run with your virus protection application. In some cases the virus protection application may even delete the application. In that case you may see that you have <APPLICATION NAME>.shim but no <APPLICATION> file. The process for whitelisting files will depend on your particular system’s virus protection application; Please see the corresponding documentation for details.

Ensuring applications are on the search path

Many of the applications in the Haskell toolchain rely on being on the shell’s search path, which is a list of directories in which applications are looked for. This list is defined by the PATH variable for most commonly-used shells. To ensure that the applications installed by e.g. GHCup are on the search path, the directories in which the applications are installed into are usually added to the PATH variable as part of a so-called startup file which is processed whenever the shell is first invoked.

To check what directories are listed on your search path, run echo $PATH in your shell and you should see a string of colon-separated directories. For example, if you are on a macOS system then you should see entries similar to /Users/username/.cabal/bin, /Users/username/.ghcup/bin, and possibly an entry similar to /opt/homebrew/opt/llvm/bin in the event that you needed to install a different version of LLVM.

If the appropriate directories are not in your current search path, try the following troubleshooting steps.

  1. Try opening a new shell to ensure that your shell has read in the most recent version of the startup files.

  2. Ensure that the required directories are added to the PATH variable in your startup files.

    1. Check which shell you are using by running the command ps -p $$.

    2. Check that the appropriate directories are added to one of the startup files for your shell.

      • If you are a Bash user then check ~/.bashrc (or whichever file you put them in)

      • If you are a Z shell user then check ~/.zshrc (or whichever file you put them in).

      • The entries should be of the form export PATH="/Users/username/.ghcup/bin:$PATH" (the paths will by system and username specific).

  3. Ensure that the SHELL variable corresponds to the shell that you are using.

    1. Run echo $SHELL to display the value of SHELL.

    2. If you are a Bash user and SHELL isn’t appropriately set then run the command chsh -s /bin/bash.

    3. If you are a Z shell user and SHELL isn’t appropriately set then run the command chsh -s /bin/zsh.

  4. Ensure that the application you are using to launch the shell is configured to launch the appropriate shell. Many applications use the value of SHELL to determine which shell to use, but if that isn’t working you can often specify the desired shell. To configure VSCode’s terminal see the Integrated Terminal instructions.

SSL certificate error message

If you receive an error with a message like the following:

curl: (60) SSL certificate problem: self signed certificate in certificate chain

please try the following workarounds:

  • Upgrade the version of curl. If you run curl --version and it lists an old release date, then you may want to install a newer version. You can see when your version of curl was released by viewing the curl - Release web page.

  • Update your OpenSSL certificates. If you installed OpenSSL on macOS through Homebrew, then you can use the command brew postinstall openssl. If you did not use Homebrew, you can install OpenSLL from source.

  • Use wget instead of curl when running GHCup by specifying the option --downloader=wget. So for example, you can use the command ghcup --downloader=wget list

IDE-Specific Guidance

The purpose of this section is to provide IDE-specific instructions for creating the optimal Haskell experience.

At this time, we are only providing guidance on Visual Studio Code. Other IDEs such as Emacs and Vim may be optimal for certain users, but require more configuration. We hope to include guidance on these IDEs in the future.

Visual Studio Code

Visual Studio Code has many extensions to support Haskell development. The recommended extensions are listed below.

Note that haskell-linter requires some additional work.

Every extension can be configured by navigating to Code > Preferences > Settings and selecting "Extensions" from the menu. We recommend the following Settings:

  • Haskell

    • Set "Formatting Provider" to "brittany"

Nix

MacOS Example

The following works as of nix version 2.10.3.

  1. Install the nix package manager.

  2. Configure nix by modifying and/or adding the following files.

    /etc/nix/nix.conf
    build-users-group = nixbld
sandbox = false
experimental-features = nix-command flakes
substituters = https://cache.nixos.org s3://nsstatdev-main-usae1-nix-cache https://hydra.iohk.io https://cache.iog.io https://cache.dhall-lang.org https://dhall.cachix.org 
trusted-public-keys = cache.nixos.org-1:6NCHdD59X431o0gWypbMrAURkbJ16ZPMQFGspcDShjY= nsstatdev-main-usae1-nix-cache-1:3qt3qqlXhyl2HGK8UE1Eh12NEmoyK8mx81uDWDAKPn4= hydra.iohk.io:f/Ea+s+dFdN+3Y/G+FDgSq+a5NEWhJGzdjvKNGv0/EQ= cache.dhall-lang.org:I9/H18WHd60olG5GsIjolp7CtepSgJmM2CsO813VTmM= dhall.cachix.org-1:8laGciue2JBwD49ICFtg+cIF8ddDaW7OFBjDb/dHEAo=
post-build-hook = /etc/nix/upload-to-cache.sh
max-jobs = auto
    /etc/nix/key.private

    Request key on slack on stats-software-dev channel.

    /etc/nix/upload-to-cache.sh

    This is similar to the script that the nsBuild nix docker image uses.

    #!/bin/sh

set -eu
set -f # disable globbing
export IFS=' '

nix store sign --key-file /etc/nix/key.private $OUT_PATHS
echo "Uploading paths" $OUT_PATHS
exec nix copy --to 's3://nsstatdev-main-usae1-nix-cache' $OUT_PATHS

    Set this script to executable:

    chmod +x /etc/nix/upload-to-cache.sh

  3. Test out nix by running: nix run nixpkgs#hello. You should you see:

    Hello, world!

node

  1. Go to https://nodejs.org/en/download/

  2. Follow the instructions for your operating system

For ETL - asclepias integration testing:

  1. Create a personal access token at gitlab.com for your TargetRWE user account.

  2. Once you have the token, run the following commands in a terminal

---
npm config set @etl-pipeline:registry https://gitlab.com/api/v4/projects/38176586/packages/npm/
npm config set //gitlab.com/api/v4/projects/38176586/packages/npm/:_authToken [YOUR_ACCESS_TOKEN]
---

Python Setup

R Setup

Rust Setup

Visual Studio Code Setup

This section provides instructions for how to set up Visual Studio Code. This does not include any language-specific instructions. For language-specific instructions, see the setup instructions for that language.

Instructions

  • Download Visual Studio Code here.

  • Download appropriate Extensions.

  • Clone project repos. See GitLab Setup for instructions.

    • You may need to set up a Personal Access Code. See instructions here.

    • If using GitLab Worflow extension:

Extensions

Listed below are VS Code extensions that may be useful in general. For language-specific extensions, see the usage guides.

  • GitLab Workflow

  • AWS Toolkit

  • PowerShell

  • AsciiDoc