Font size
Site color
Skip to main content
  • Step-by-step guide

    A minimal configuration


    We will use the fictional repository https://github.com/myorg/myrepo. The following steps set up the minimum layout:

    1. Clone the repo

    $ export MY_REPO=https://github.com/myorg/myrepo
    $ git clone $MY_REPO
    
    1. A good practice is to add the changes herein described in an individual branch, so not directly in the production –usually master– branch. In this example we will use setup_jenkins-pipeline-library as follows:

    $ cd $MY_REPO
    $ git checkout -b setup_jenkins-pipeline-library
    
    1. We will now create the following directory structure within the code repository, as required by the library (be sure to read Layout section):

      |-- .sqa
      |    |-- config.yml
      |    |-- docker-compose.yml
      |-- Jenkinsfile
      

      3.1. Create the main .sqa folder:

      $ mkdir .sqa
      

      3.2. Create the initial content of the main configuration file, .sqa/config.yml, with the description of your repository (see The configuration file: config.yml section):

      $ cat <<EOF > .sqa/config.yml
      config:
        project_repos:
          myrepo:
            repo: 'https://github.com/myorg/myrepo'
      EOF
      

      3.3. Create the .sqa/docker-compose.yml. For the time being, we will only specify the version required by the library (i.e. 3.6), later on we will add the service definitions (see The services: docker-compose.yml section):

      $ cat <<EOF > .sqa/docker-compose.yml
      version: "3.6"
      EOF
      

      3.4. In the root path of the code repository, create the Jenkinsfile, file required by Jenkins. In order to make it work with the jenkins-pipeline-library, at least the following content must be present (see The pipeline: Jenkinsfile section):

      $ cat <<EOF > Jenkinsfile
      @Library(['github.com/indigo-dc/jenkins-pipeline-library@release/2.1.0']) _
      
      def projectConfig
      
      pipeline {
          agent any
      
          stages {
              stage('SQA baseline dynamic stages') {
                  steps {
                      script {
                          projectConfig = pipelineConfig()
                          buildStages(projectConfig)
                      }
                  }
                  post {
                      cleanup {
                          cleanWs()
                      }
                  }
              }
          }
      }
      EOF
      
    2. Commit & push the layout files:

    $ git add .sqa Jenkinsfile
    $ git commit -m "Initial setup of jenkins-pipeline-library files"
    

    Now that we have the skeleton with an initial version of the three relevant files, let’s add our checks through the sqa_criteria setting. We will see how to do that in the next section.



    The sqa_criteria setting


    In this section we will cover the sqa_criteria setting, which represents the fundamental part of the configuration since it contains the definitions of the checks that comprise the quality criteria.

    The full set of criteria currently supported in the library is summarized in the following table and can be found in the sqa_criteria section.

    sqa_criteria setting

    What does it cover?

    qc_style

    Make your code compliant with a style standard

    qc_coverage

    Calculate the unit testing coverage of your code

    qc_functional

    Test the main features of your software

    qc_security

    Assess the security (uncover vulnerabilities & bad security practices)

    qc_doc

    Generate the documentation


    Python and Java examples

    The best way to learn the basics about the library is through examples. In the next subsections we will show how to fulfill the qc_style criterion for Python and Java applications. Working configurations are provided so you can readily test them.

    The current version of the library supports both commands and tox (this last only available for Python-based applications) environments to execute the checks, hence the examples below cover the three possible use cases, i.e. Python (with and without Tox) and Java (with commands).


    Python with tox

    config.yml

    config:
      project_repos:
        myrepo:
          repo: 'https://github.com/myorg/myrepo'
    
     sqa_criteria:
       qc_style:
         repos:
           myrepo:
             container: myrepo-testing
             tox:
               tox_file: /myrepo-testing/tox.ini
               testenv: stylecheck
    
    docker-compose.yml

    version: "3.6"
    
    services:
      myrepo-testing:
        image: "indigodatacloud/ci-images:python3.6"
        container_name: "myrepo-testing"
        volumes:
         - type: bind
           source: ./myrepo
           target: /myrepo-testing
    
    tox.ini

    [tox]
    minversion = 2.1
    envlist = py{36,37},stylecheck
    skipsdist = True
    
    [testenv]
    usedevelop = True
    basepython = python3
    
    [testenv:stylecheck]
    envdir = {toxworkdir}/shared
    commands = flake8
    

    As it can be seen, the three files are linked together. In order to compose
    them, the following requirements must be considered:
    Notes on docker-compose.yml (DC)
    1. Minimum version: 3.6 [DC] is required, otherwise bind volume definitions are not correctly supported.

    2. There are 3 main parameters that must be defined in DC file, i.e.:

      • container_name: sets the name of the service.

      • image: points to the Docker image that will be used by the container.

        • When using this parameter, the image must be previously available in Docker Hub registry. Additionally, DC allows the image to be built in runtime by providing a Dockerfile. In this case, the build parameter must be used (check out DC’s build parameter documentation).

        • Note that all the tools required to run the tests must be deployed in the Docker image. In this example, the indigodatacloud/ci-images:python3.6 image already contains the tools needed to execute the subsequent tox commands.

        • Last but not least, you should check whether the image in use is configured to run-and-die (*one-shot*). If this is the case, add a sleep infinity command in the DC’s command parameter, as explained in DC’s section Summary of recommendations for best use with the library.

      • volumes: identifies the volume where the repository (myrepo in this example) content will be accessible. The type: bind is required and only the values for source and target parameters must be provided.

    Notes on links between config.yml (CONFIG) and docker-compose.yml (DC)
    1. The value for the container setting [CONFIG] must correspond to a service definition in the DC file. In the example above, the service myrepo-testing is defined under services inside DC file.

    2. The source parameter [DC file] corresponds to the ID/name used to identify the current repository, i.e. the ID used in the config:project_repos definition [CONFIG]. Since we are using a relative path in the DC file specification, the source [DC file] value must always be prefixed by ./ (DC always expect a path format). In our example, we have set myrepo as the ID so the correct value for source [DC file] is ./myrepo.

    Notes on links between tox.ini (TOX), config.yml (CONFIG) and docker-compose.yml (DC)
    1. The value for tox_file [CONFIG] must be the absolute path to the TOX file. To obtain the full path to the TOX file, target [DC file] must be prepended to the relative path of the TOX file within the code repository, as it is the folder where the repository has been checked out. In the example above, myrepo has the TOX file available in the root path of the repository, therefore /myrepo-testing/tox.ini is the correct location.

    2. The value for testenv [CONFIG] must correspond to any of the test environments [TOX file]. In our example, stylecheck testenv executes the flake8 style tool, and thus, it can be used as the value for tox’s testenv [CONFIG].

    Tip

    We recommend the use of Tox tool in the case of Python applications, as it is the most accurate way of defining and running all your tests. Hence, Tox can execute each test in an individual Python virtual environment (virtualenv), so it is isolated from the other tests. Note that the use of Tox in this example is extremelly simple and does not take advantage of the full capabilities of the tool.


    Python with commands

    config.yml
    config:
      project_repos:
        myrepo:
          repo: 'https://github.com/myorg/myrepo'
    
     sqa_criteria:
       qc_coverage:
         repos:
           myrepo:
             container: myrepo-testing
             commands:
               - flake8
    

    docker-compose.yml
    version: "3.6"
    
       services:
         myrepo-testing-java:
           image: "indigodatacloud/ci-images:java"
           container_name: "myrepo-testing-java"
           volumes:
            - type: bind
              source: ./myrepo
              target: /myrepo-testing

    In this example, the only difference with respect to the previuos example is the use of commands [CONFIG]. Here, we will obtain the same output as in the previous Python-with-tox example since flake8 tool is executed.


    Java with commands

    config.yml
    config:
      project_repos:
        myrepo:
          repo: 'https://github.com/myorg/myrepo'
    
     sqa_criteria:
       qc_coverage:
         repos:
           myrepo:
             container: myrepo-testing-java
             commands:
               - mvn -f /myrepo-testing/pom.xml checkstyle:checkstyle
    
    docker-compose.yml
    version: "3.6"
    
       services:
         myrepo-testing-java:
           image: "indigodatacloud/ci-images:java"
           container_name: "myrepo-testing-java"
           volumes:
            - type: bind
              source: ./myrepo
              target: /myrepo-testing

    Don’t forget to commit

    Once you have added one of the former definitions in the sqa_criteria setting, it is time to commit our work. Following up with the example of previous section:

    $ git commit -m "Add sqa_criteria setting & associated docker-compose services"
    

    In the next section, we will provide the last steps to make all this work being executed in Jenkins.



    Last steps


    After following the previous steps, we have a working example that validates the code style of our Python or Java application.

    Throughout the guide we have learnt that:

    • There are 3 required files (provided with relative paths to the code repository)

      • .sqa/config.yml

      • .sqa/docker-compose.yml

      • Jenkinsfile

      Apart from those, in the specific case of using Tox tool with Python, we also need to consider both the definition of the environments and the location of the Tox configuration file.

    • The checks are defined within the sqa_criteria setting of the .sqa/config.yml file.

    • The services are provided through Docker containers, using Docker Compose solution, and must be previously defined in the .sqa/docker-compose.yml file.

    Now it is time to push our changes to the remote repository:

    $ git push origin setup_jenkins-pipeline-library
    

    If we have a Jenkins server already scanning our remote repository, the previous push command will automatically trigger the execution of the code style check we have defined in our example. No need of doing anything else. Once the check has been executed by Jenkins, the results will be displayed. The setup_jenkins-pipeline-library branch will be eventually merged into the production branch (usually master) after the dealing with the style standard complaints (if any).

    We provide a working example with Jenkins build logs available.

    The EOSC-Synergy project provides a Jenkins instance that can be used for research software projects using the jenkins-pipeline-library (v2). If you need support, please contact <wp3@eosc-synergy.eu>.


    JePL Practical Demonstration