Explained: Executer Stage

Behaviour

During the Executer stage, SQL --previously created by the Templater stage-- is executed to one of the DTAP environments. More precisely, only the files that are changed in the preceding Templater commit are executed in full. We refer to the contents of these files as the actual change. The deployment order is as follows: → Test → Acceptance → Production. Note that every arrow requires a manual approval; a users needs to confirm that the next environment is ready for the change.

manual-approve
Figure 1. Manual approve step in deployment pipeline

A Deployment is performed by the Executer as follows. First, the Executer determines the files involved in the Change. Next, these files are executed from top to bottom and in alphabetical order. From this precedure, it follows that SQL files need to be designed with multiple execution in mind. Otherwise, an execution of a full file with a minor change would not be succesful.

If an error occurs during the Deployment, the Executer logs the error and skips the remaining statements in the file it is executing. It continues with the next file if one exists as shown below. Numbers illustrate the order of execution.

change with error
Figure 2. Order of execution in case of error

Execution logs of a Deployment are added to the repository after the Deployment is complete. The Executer creates one execution log per SQL file it executed. An execution log is the original executed SQL source annotated with a color. Green colored SQL statements are executed succesfully, visa versa for red colored statements. NOTE: At most one red statement can be found at the end of an SQL file, because the executer skips statements after a failing statement

Order of deployment

ADM guarantees that when conflicting changes deployed similtaniously, a risk is introduced that tests are invalid.

It takes time to deploy a change first to Development, followed by Test, followed by Acceptance to finally be deployed in Production. When multiple changes are simultaniously in transit, one change can overtake another, introducing the risk that a test can be made invalid. Take the following example as illustration:

alt: order of execution
Figure 3. Importance of execution-order
1 Say that if C1 is followed by C2, both executions are succesfull
2 In case C2 changes a prerequisite of C1, a change in order will cause C1 to fail
3 In case C2 is dependent on C1, a change in order will cause C2 to fail

The above mentioned cases can occur on Acceptance in the following scenario:

  • Say that C1 is deployed on Test (short for Test environment)

  • Next, C2 is deployed on Test.

  • If C2 will be deployed before C1 to Acc, the order is inverted on Acceptance,

Our solution is simple: before any change is deployed by the Executer, we execute to the DryRun environment first. This will ensure that if the order of execution is inverted and an error would occur, that this error would be caught in the DryRun environment, enabling the developer to make a fix.

FAQ

What to do when a failure occurs

  • ReExecute: Some error are fixed with a ReRun

    • A disabled ADM-user will not be able to execute. The fix: (1) enable the user, (2) initiate a rerun from the Git-CI/CD-webconsole

    • A network timeout to a cloud database can occur because of a network failure. The fix: (1) verify that the Cloud-database is reachable from the ADM-container, (2) intiate a rerun from the Git-CI/CD-webconsole