Introduction

Double Open aims to automate open source compliance for the Yocto Project.

TLDR

  1. Add meta-doubleopen layer to Yocto's conf/bblayers.conf.

  2. Add INHERIT += "doubleopen" to Yocto's conf/local.conf.

  3. Build the image with Yocto. The resulting SPDX will be output to DEPLOY_DIR_IMAGE as <IMAGE_NAME>.spdx.json.

  4. Use CLI to upload missing packages to Fossology:

    doubleopen fossology -u <FOSSOLOGY_API_URI> -t <FOSSOLOGY_TOKEN> \
       upload \
       -s <SPDX_DEPLOY_DIR>/*.tar.bz2 \
       -f <FOSSOLOGY_FOLDER_ID> \
       --spdx <IMAGE_SPDX>
    
  5. Use CLI to populate the SPDX file with license data:

    doubleopen fossology -u <FOSSOLOGY_API_URI> -t <FOSSOLOGY_TOKEN> \
       query \
       -i <IMAGE_SPDX> \
       -o <OUTPUT_SPDX>
    
  6. Convert the SPDX file to ORT's format:

    orth convert-spdx-to-ort \
       -i <INPUT_SPDX> \
       -o <OUTPUT_ORT_FILE> \
       --repository-configuration-file <ORT.YML>
    
  7. Clone ORT configuration and policy.

  8. Evaluate:

    ORT_CONFIG_DIR=<POLICY_DIR> \
    ort evaluate \
       -i <ORT_RESULT> \
       -o <EVALUATOR_RESULT_DIR> \
    
  9. Generate reports and notices:

    ORT_CONFIG_DIR=<POLICY_DIR> \
    ort report \
       -i <EVALUATOR_RESULT_DIR>/evaluation-result.yml \
       -o <REPORT_DIR> \
       -f <FORMATS>
    

Installation

Double Open CLI can be downloaded from GitHub releases.

We recommend using OSS Review Toolkit with Docker. Image for our fork of ORT can be found from DockerHub.

Setup Yocto

Double Open's layer, meta-doubleopen is used to create an SPDX Document describing the image built with Yocto. The layer needs to be added to conf/bblayers.conf and enabled with INHERIT += "doubleopen" in conf/local.conf.

After meta-doubleopen is added, bitbake build <RECIPE> produces SPDX Documents for all packages in <SPDX_DEPLOY_DIR> (defaults to /build/tmp/deploy/spdx/) and the SPDX Document for the whole image is saved to DEPLOY_DIR_IMAGE.

Get data from Fossology

Upload source

For Fossology to be able to return licensing and copyright data for the source files, they need to be uploaded to Fossology. The source archives to upload are filtered to exclude proprietary packages from being scanned by not uploading source archive for the recipe if the license of the recipe includes CLOSED.

Uploading is done with Double Open CLI with the following command:

doubleopen fossology -u <FOSSOLOGY_API_URI> -t <FOSSOLOGY_TOKEN> \
  upload \
  -s <SPDX_DEPLOY_DIR>/*.tar.bz2 \
  -f <FOSSOLOGY_FOLDER_ID> \
  --spdx <IMAGE_SPDX>

Populate SPDX

After the files have been scanned in Fossology, the SPDX Document of the image can be populated with the data from the Fossology API. The API is queried with tha SHA256 hash values of the source files. This is done with Double Open CLI with the following command:

doubleopen fossology -u <FOSSOLOGY_API_URI> -t <FOSSOLOGY_TOKEN> \
  query \
  -i <IMAGE_SPDX> \
  -o <OUTPUT_SPDX>

Conversion

The conversion from SPDX to ORT's data format is done with command in our fork of ORT. The command is part of the ORT's Helper CLI:

orth convert-spdx-to-ort \
  -i <INPUT_SPDX> \
  -o <OUTPUT_ORT_FILE> \
  --repository-configuration-file <ORT.YML>

The command creates a file mimicking ORT's scanner result which can be used to Evaluate the image's license compliance with ORT's Evaluator and to create reports with ORT's Reporter.

To enable different rules for scanner findings and concluded licenses from Fossology, scanner findings are identified with LicenseRef-Scanner- prefix. This is not needed for notice generation, so the conversion for notice generation should be done with the following command:

orth convert-spdx-to-ort \
  -i <INPUT_SPDX> \
  -o <OUTPUT_ORT_FILE> \
  --repository-configuration-file <ORT.YML> \
  --skip-scan

If your policy does not require different rules for scanner results and concluded licenses, the evaluation can be performed with the --skip-scan conversion.

Evaluate

OSS Review Toolkit is used to Evaluate the image's license compliance against a policy. The policy is defined with a license classification grouping licenses in categories and a rules script file.

The license classifications file and rules script file should be stored in a configuration directory for ORT and be named license-classifications.yml and rules.kts respectively. This enables ORT to use them automatically without having to specify them separately.

ORT_CONFIG_DIR=<POLICY_DIR> \
ort evaluate \
  -i <ORT_RESULT> \
  -o <EVALUATOR_RESULT_DIR> \

License classifications

We maintain a license classifications file at Double Open's Policy Configuration repository.

Rules

An example of the rules file can be found in the ORT repository.

Generate notices

Notice files for the images are generated with ORT with the following command:

ORT_CONFIG_DIR=<POLICY_DIR> \
ort report \
  -i <EVALUATOR_RESULT_DIR>/evaluation-result.yml \
  -o <REPORT_DIR> \
  -f <FORMATS>