How-to: setup Github repository

In this how-to, we setup an existing Github repostory to with ADM. At the end of this how-to-guide, you will have an operational actions workflow with which you can provision your Snowflake cloud database.

We will use the following technologies

Technologies

  1. a Github-repository

  2. Github workflow file, that holds the configuration for CI/CD

  3. add members/owners to the repository

Prerequisites

  1. Have an empty Github repository where we can setup an ADM-configuration.

Connect Snowflake to ADM

  1. Clone the repository to your local machine

  2. Add the file environments/development.yml and fill it with the following contents. This will create an ADM-environment called development that ADM uses to connect to Snowflake.

Listing 1. Setup an ADM-environment called development
{
  "name": "development",(1)
  "description": "Snowflake environment called 'development'.",
  "connector": "snowflake",
  "security": {
    "connector": "variables",
    "endpoint_variable": "DEVELOPMENT_SNOWFLAKE_ENDPOINT",
    "username_variable": "DEVELOPMENT_SNOWFLAKE_USER",
    "password_variable": "DEVELOPMENT_SNOWFLAKE_PASS"
  }
}

Setup secrets

Next, we assign values to the environment variables that we defined in the previous step.

  1. Login to Github, and go to your repository.

  2. Go to menu:Settings[Secrets > New Repository Secret]and add the following variables.

Variable Key Variable Value Definition

DEVELOPMENT_SNOWFLAKE_ENDPOINT

development.eu-west-1

Name of the Snowflake account

DEVELOPMENT_SNOWFLAKE_USER

SF_ACHERON_AUTO_EXECUTER

The name of the Snowflake user created earlier

DEVELOPMENT_SNOWFLAKE_PASS

<developments-snowflake-password>

The password of the Snowflake user created earlier

SSH_PRIVATE_KEY

…​

SSH private key that will be used for Git actions like pull and commit. Supported formats are: ed25519 and RSA
Example screen Github Secrets
Figure 1. Example Screenshot of variables screen

Create actions workflow

A preconfigured github workflow yml holds the configuration of the Github actions. Add it to the .github/workflow folder of your repository.

Create actions workflow, use ADM

name: Acheron Database Manager CI/CD
on: [push]
jobs:
  linter:(2)
    runs-on: ubuntu-latest
    container:
      image: registry.gitlab.com/acheron-it/docker/acheron-database-manager/image:*.*.*-*.*.* (1)
      credentials:
          username: user
          password: ${{ secrets.docker_hub_password}}
    steps:
      - name: Checkout
        uses: actions/checkout@v1
      - name: Run linter step (4)
        shell: bash
        run: |
          echo "Linting ..."
          python /usr/local/lib/python3.8/site-packages/opt/acheron/lint.py

  templater:(2)
    needs: linter
    runs-on: ubuntu-latest
    container:
      image: registry.gitlab.com/acheron-it/docker/acheron-database-manager/image:*.*.*-*.*.* (1)
      credentials:
        username: user
        password: ${{ secrets.docker_hub_password}}
    env:
      ACH_DEPLOY_USERNAME: ${{ secrets.ACH_DEPLOY_USERNAME }}
      SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
      CI_REPOSITORY_URL: "git@github.com:rdaalman-ach/acheron-adm-demo-klant.git"
      ACHERON_GIT_PORT: 22
      GIT_COMMIT_COMMITTER_EMAIL: "rob@acheron.cloud"
    steps:
      - name: Run templater step (5)
        shell: bash
        run: |
          echo "Templating ...."
          eval $(ssh-agent -s)
          python /usr/local/lib/python3.8/site-packages/opt/acheron/template.py "$@" development

  executer:(2)
    needs: templater
    runs-on: ubuntu-latest
    container:
      image: registry.gitlab.com/acheron-it/docker/acheron-database-manager/image:*.*.*-*.*.* (1)
      credentials:
          username: user
          password: ${{ secrets.docker_hub_password}}
    env:
      ACHERON_SNOWSQL_ENDPOINT: ${{ secrets.ACHERON_SNOWSQL_ENDPOINT }}
      ACHERON_SNOWSQL_PASS: ${{ secrets.ACHERON_SNOWSQL_PASS }}
      ACHERON_SNOWSQL_USER: ${{ secrets.ACHERON_SNOWSQL_USER }}
    steps:
      - name: Checkout
        uses: actions/checkout@v1
      - name: Run executer (6)
        shell: bash
        run: |
          echo "Executing to Snowflake"
          python /usr/local/lib/python3.8/site-packages/opt/acheron/execute.py "$@" development

Create actions workflow, use ADM with predefined Github actions

name: Acheron Database Manager CI/CD
on: [push]
jobs:(2)
  linter:(3)
    runs-on: ubuntu-latest
    container:
      image: registry.gitlab.com/acheron-it/docker/acheron-database-manager/image:*.*.*-*.*.* (1)
      credentials:
          username: user
          password: ${{ secrets.docker_hub_password}}
    steps:
      - name: Checkout
        uses: actions/checkout@v1
      - shell: bash
        run: shopt -s expand_aliases
      - shell: bash
        run: source /etc/aliases.sh
        working-directory: /
      - name: Run linter step
        shell: bash
        run: |
          echo "Linting ..."
          python /usr/local/lib/python3.8/site-packages/opt/acheron/lint.py

  templater:(4)
    needs: linter
    runs-on: ubuntu-latest
    container:
      image: registry.gitlab.com/acheron-it/docker/acheron-database-manager/image:*.*.*-*.*.* (1)
      credentials:
        username: user
        password: ${{ secrets.docker_hub_password }}
    env:
      ACH_DEPLOY_USERNAME: ${{ secrets.ACH_DEPLOY_USERNAME }}
      SSH_PRIVATE_KEY: ${{ secrets.SSH_PRIVATE_KEY }}
      ACHERON_GIT_PORT: 22
      GIT_COMMIT_COMMITTER_EMAIL: "user@acheron.cloud"
    steps:
      - name: Setup ssh
        uses: webfactory/ssh-agent@v0.5.1
        with:
          ssh-private-key: ${{ secrets.SSH_PRIVATE_KEY }}
      - name: Checkout
        uses: actions/checkout@v1
      - name: Run templater step
        shell: bash
        run: |
          echo "Templating ..."
          python /usr/local/lib/python3.8/site-packages/opt/acheron/template.py "$@" development --delegate-repo-pull --delegate-repo-push --input-dir=config
      - run: git add .
      - name: Push changes
        uses: actions-js/push@master
        with:
          github_token: ${{ secrets.GITHUB_TOKEN }}
          message: Commit new Container run result

  executer:(5)
    needs: templater
    runs-on: ubuntu-latest
    container:
      image: registry.gitlab.com/acheron-it/docker/acheron-database-manager/image:*.*.*-*.*.* (1)
      credentials:
          username: user
          password: ${{ secrets.docker_hub_password}}
    env:
      ACHERON_SNOWSQL_ENDPOINT: ${{ secrets.ACHERON_SNOWSQL_ENDPOINT }}
      ACHERON_SNOWSQL_PASS: ${{ secrets.ACHERON_SNOWSQL_PASS }}
      ACHERON_SNOWSQL_USER: ${{ secrets.ACHERON_SNOWSQL_USER }}
    steps:
      - name: Checkout
        uses: actions/checkout@v1
      - name: Run executer
        shell: bash
        run: |
          echo "Executing to Snowflake"
          python /usr/local/lib/python3.8/site-packages/opt/acheron/execute.py "$@" development
1 Fill in the exact version number of the Acheron ADM container that you’re using. In case this is unclear, feel free to contact Acheron (info@acheron.cloud)
2 The jobs that will be executed every time that a change is pushed to Github - Github jobs
3 Lint-step evaluates the input files
4 Template-step renders templates to SQL-files
5 Execution, executes SQL-changes on the desired Snowflake Account

Run the actions workflow for the first time

  1. Create a new branch

  2. Add the ADM-license to the environments-directory

  3. Add templates to the templates-directory

  4. Add a template-configuration to the input directory

  5. See that the linter is working

  6. Merge the branch

    1. This will trigger the Templater to make a commit

    2. The executer will execute your code to SF dryrun environment

    3. The executer will execute your code to final SF environment