How-to: setup Jenkins CI/CD for a git repository

In this how-to, we setup an existing git repository to with ADM on Jenkins. At the end of this how-to-guide, you will have an operational Jenkins pipeline with which you can provision your Snowflake cloud database.

We will use the following technologies

Technologies

  1. Settings Jenkins

  2. Jenkins Pipeline

Prerequisites

  1. Have an empty Git repository where we can setup an ADM-configuration.

  2. Have a running Jenkins setup

Connect Snowflake to ADM

  1. Clone the git repository to your local machine

  2. Add the file environments/development.yml and fill it with the following contents. This will create an ADM-environement called development that ADM uses to connect to Snowflake.

Listing 1. Setup an ADM-environment called development
{
  "name": "development",(1)
  "description": "Snowflake environment called 'development'.",
  "connector": "snowflake",
  "security": {
    "connector": "variables",
    "endpoint_variable": "DEVELOPMENT_SNOWFLAKE_ENDPOINT",
    "username_variable": "DEVELOPMENT_SNOWFLAKE_USER",
    "password_variable": "DEVELOPMENT_SNOWFLAKE_PASS"
  }
}

Setup variables

Next, we assign values to the environment variables that we defined in the previous step.

  1. Go to your Git repository and open the Jenkinsfile.

  2. Add the following variables to your pipeline.

---
 environment {
        SSH_PRIVATE_KEY     = credentials('ssh_id')
        ACH_DEPLOY_USERNAME = 'acheron_container_rw_token'
        CI_REPOSITORY_URL = 'gitlab.com/acheron-it/docker/snowflake-templater-jenkins-demo.git'
        AY68407_SNOWSQL_PASS = credentials('SNOWSQL_PASS')
        AY68407_SNOWSQL_REGION = 'eu-west-1'
        AY68407_SNOWSQL_USER = credentials('SNOWSQL_USER')
    }
---

For more information: Jenkins using environment variables

Create Pipeline

A Jenkinsfile with Pipeline setup is descibed, it holds the configuration of the Jenkins Pipeline steps. Add it to the root folder of your repository.

pipeline {
    agent {
        docker { (1)
            image 'registry.gitlab.com/acheron-it/docker/acheron-snowflake-templater/image:latest'
            args '-v ${ACH_DEPLOY_USERNAME} -v ${SSH_PRIVATE_KEY} -v ${CI_REPOSITORY_URL} -v ${AY68407_SNOWSQL_PASS} -v ${AY68407_SNOWSQL_REGION} -v ${AY68407_SNOWSQL_USER}'
            args '-u root:root -v /var/lib/jenkins/workspace/myworkspace:/tmp/' + ' -v /var/lib/jenkins/.ssh:/root/.ssh'
        }
    }
    environment { (2)
        SSH_PRIVATE_KEY     = credentials('ssh_id')
        ACH_DEPLOY_USERNAME = 'acheron_container_rw_token'
        CI_REPOSITORY_URL = 'gitlab.com/acheron-it/docker/snowflake-templater-jenkins-demo.git'
        AY68407_SNOWSQL_PASS = credentials('SNOWSQL_PASS')
        AY68407_SNOWSQL_REGION = 'eu-west-1'
        AY68407_SNOWSQL_USER = credentials('SNOWSQL_USER')
    }
    stages { (3)
        stage('Lint') { (4)
            steps {
                sh 'echo username ${ACH_DEPLOY_USERNAME}'
                sh 'echo ${SSH_PRIVATE_KEY}'
                sh 'echo repository url ${CI_REPOSITORY_URL}'
                sh 'echo "Linting ..."'
                sh 'python /lint.py'
            }
        }

        stage('Template') { (5)
            when {
                branch 'master'
            }
            steps {
                sh 'ssh-keyscan -H gitlab.com >> ~/.ssh/known_hosts'
                sh 'echo "Templating ..."'
                script {
                    def acheron_start = sh(returnStdout: true, script: '/acheron_start.sh').trim()
                    println("acheron_start: ${acheron_start}")
                }
            }
        }

        stage('Execute') { (6)
            when {
                beforeInput true
                branch 'master'
            }
            input {
                message "Deploy to production?"
                id "simple-input"
            }
            steps {
                sh 'echo "Starting snowflake execution"'
                sh 'ls'
                sh 'python ay68407.py'
            }
        }
    }
}
1 Fill in the exact version number of the Acheron ADM container that you’re using. In case this is unclear, feel free to contact Acheron (info@acheron.cloud)
2 Set environment variables
3 The stages that will be executed every time that a pipeline is started - Jenkins Pipeline
4 Lint-step evaluates the input files
5 Template-step renders templates to SQL-files
6 Execution, executes SQL-changes on the desired Snowflake Account

Run the Pipeline

  1. Create Jenkinsfile in your git repository

  2. Load the license

  3. Add a template

  4. Add a configuration

  5. Login to the Jenkins environment

  6. Select created ADM pipeline

  7. See that the linter is working

  8. Merge the branch

    1. This will trigger the Templater to make a commit

    2. The executer will execute your code to SF dryrun environment

    3. The executer will execute your code to final SF environment