The execution of security tests in the CI/CD pipelines - example with Gauntlt [ENGLISH]

Gauntlt (“Be Mean to Your Code”) allows you to run security tests in the CI/CD pipelines. Specifically, Gauntlt can be used:

  • in Continuous Integration, by running Security Smoke Tests: these are quick, simple tests,
  • In Continuous Delivery, by automating attack simulations.

If tests fail, the team must then immediately fix the identified problems.

The implementation of these security tests must be done in a separate flow from the pipeline used by the developers, so as not to block their builds. If the security tests fail, the results must be reviewed. At the end of this review, the failure threshold can be modified to avoid the appearance of too many false positives.

Once all that has been stabilized, you can integrate these tests into the build flow and block the build if the failure threshold is exceeded.

Gauntlt is based on Cucumber, a Behavior-driven development (BDD) testing framework based on Ruby. The use of BDD simplifies the writing of security and control tests of the application and its configuration.

It contains attack adapters that mask the technique needed to use these PenTest tools. Examples of attack files are:

  • SSL configuration check (using sslyze),
  • Network configuration check (using nmap),
  • Testing SQL injection vulnerabilities (using sqlmap),
  • Performing simple web application attacks (using curl),
  • Search for common vulnerabilities (using arachni, dirb and garmr),
  • Checking for specific vulnerabilities (e.g. Heartbleed).

These attack files are provided as examples. They can be used directly, after changing the arguments, but you can customize them to create new attacks.

The attack files are described in a Ruby DSL (Domain Specific Language), called Gherkin. Gherkin uses the following syntax:

Given {preconditions}
When {execution of the test steps}
Then {expected results}

Since each test returns a pass or fail result, it is simple to integrate these tests into a CI or CD pipeline.

Let’s take an example of an attack file. These files have a “.attack” suffix. Each file therefore contains one or more scenarios, each containing several Given / When / Then steps:

Feature: Description of the attack or of the verification
Background: Setting the context for all scenarios
Scenario: Logic of the attack in Given/When/Then format
Given “tool” is installed
When I launch a “tool” attack with:
    Then it should pass with/should contain/and should not contain:

Here is a complete and real example, which consists in searching for XSS (Cross Site Scripting) on the website (which is used for security tests) with the arachni tool, integrated in Gauntlt:

Feature: Look for cross site scripting (xss) using arachni against
Scenario: Using arachni, look for cross site scripting and verify no issues are found
 Given "arachni" is installed
 And the following profile:
   | name        | value             |
   | url         |    |
 When I launch an "arachni" attack with:
 arachni --checks=xss --scope-directory-depth-limit=1 <url>
 Then the output should contain "0 issues were detected."

Now let’s see how to install Gauntlt. We will see two methods. The first one is the classical method, using the Gem package manager of Ruby :

gem install gauntlt

To launch an attack, simply issue this command:

gauntlt attacks/my_attack.attack

To launch all the attacks, the syntax is simplified:


You can also easily view the steps used in the attacks:

gauntlt --steps

To get the predefined attacks, which only need to be customized, download the source code:

git clone
cd gauntlt/examples

And to get the list of defined attacks:

gauntlt --list

The second method is to use Docker. Start by cloning the repository :

git clone

Next, build the Docker container:

make build

To verify that everything has worked properly, simply enter this command:

make help

You must then copy a binary Stub in the path of your system. To do this, run this command:

make install-stub

Let’s check that this operation has worked by running this command:

gauntlt-docker --help

Now you can create the attack files in a directory, for example the subdirectory ./attacks. To run an attack file called ./attacks/xss.attack, enter this command:

gauntlt-docker ./examples/xss.attack

This allows you to run the local attack files directly from the Docker container.

If you need to, you can also run Docker in interactive mode, to have access to the Docker container command line:

make interactive

We saw at the beginning of the post that we can distribute the security tests on the CI and on the CD. In the attack files there are annotations that indicate whether the test is fast or slow. Specifically, if the test is slow, there is an @slow annotation at the beginning of the file. If there is no such annotation, the test is fast.

So the idea is to launch slow attacks in the Continuous Delivery pipeline like this:

gauntlt --tags @slow

And to launch the other attacks, therefore the fast ones, in the Continuous Integration pipeline:

gauntlt --tags ~@slow

So the last step is to integrate these BDD tests into your Continuous Integration and Continuous Delivery pipelines. We are not talking about how to stop the build if the tests fail.

Official website of Gauntlt :


Your browser is out-of-date!

Update your browser to view this website correctly. Update my browser now