1berry/qtbase
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
qtbase/cmake/QtPublicSbomGenerationHelpers.cmake

1184 lines
44 KiB
CMake
Raw Permalink Normal View History

CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Copyright (C) 2024 The Qt Company Ltd.
# Copyright (C) 2023-2024 Jochem Rutgers
# SPDX-License-Identifier: MIT AND BSD-3-Clause
# Helper to set a single arg option to a default value if not set.
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
function(_qt_internal_sbom_set_default_option_value option_name default)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(NOT arg_${option_name})
set(arg_${option_name} "${default}" PARENT_SCOPE)
endif()
endfunction()
# Helper to set a single arg option to a default value if not set.
# Errors out if the end value is empty. Including if the default value was empty.
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
function(_qt_internal_sbom_set_default_option_value_and_error_if_empty option_name default)
_qt_internal_sbom_set_default_option_value("${option_name}" "${default}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(NOT arg_${option_name})
message(FATAL_ERROR "Specifying a non-empty ${option_name} is required")
endif()
endfunction()
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
# Helper that returns the relative sbom build dir.
# To accommodate multiple projects within a qt repo (like qtwebengine), we need to choose separate
# build dirs for each project.
function(_qt_internal_get_current_project_sbom_relative_dir out_var)
_qt_internal_sbom_get_root_project_name_lower_case(repo_project_name_lowercase)
_qt_internal_sbom_get_qt_repo_project_name_lower_case(real_qt_repo_project_name_lowercase)
if(repo_project_name_lowercase STREQUAL real_qt_repo_project_name_lowercase)
set(sbom_dir "qt_sbom")
else()
set(sbom_dir "qt_sbom/${repo_project_name_lowercase}")
endif()
set(${out_var} "${sbom_dir}" PARENT_SCOPE)
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Helper that returns the directory where the intermediate sbom files will be generated.
function(_qt_internal_get_current_project_sbom_dir out_var)
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
_qt_internal_get_current_project_sbom_relative_dir(relative_dir)
set(sbom_dir "${PROJECT_BINARY_DIR}/${relative_dir}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(${out_var} "${sbom_dir}" PARENT_SCOPE)
endfunction()
# Helper to return the path to staging spdx file, where content will be incrementally appended to.
function(_qt_internal_get_staging_area_spdx_file_path out_var)
_qt_internal_get_current_project_sbom_dir(sbom_dir)
_qt_internal_sbom_get_root_project_name_lower_case(repo_project_name_lowercase)
set(staging_area_spdx_file "${sbom_dir}/staging-${repo_project_name_lowercase}.spdx.in")
set(${out_var} "${staging_area_spdx_file}" PARENT_SCOPE)
endfunction()
# Starts recording information for the generation of an sbom for a project.
# The intermediate files that generate the sbom are generated at cmake generation time, but are only
# actually run at build time or install time.
# The files are tracked in cmake global properties.
function(_qt_internal_sbom_begin_project_generate)
set(opt_args "")
set(single_args
OUTPUT
CMake: Make ninja 'sbom' work for top-level builds We need to pass all dirs where sboms will be generated, for external document referencing. We also need to set up dependencies between the repo sbom custom targets, so that qtsvg sbom is only run after qtbase sbom. We use the dependencies.yaml info to set up the dependencies. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id3331e11742bc2c86e7ed52ce26b3ff21eace359 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-20 19:21:53 +02:00
OUTPUT_RELATIVE_PATH
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
LICENSE
COPYRIGHT
DOWNLOAD_LOCATION
PROJECT
CMake: Allow adding content to the SBOM repo project comment Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I810814afa69ddf6dc89e834f525b62d87c3aa109 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-17 16:44:31 +02:00
PROJECT_COMMENT
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
PROJECT_FOR_SPDX_ID
SUPPLIER
SUPPLIER_URL
NAMESPACE
CPE
OUT_VAR_PROJECT_SPDX_ID
)
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Add a fake deterministic SBOM generation option Replaces content like timestamps, version strings and checksums with constant values to make the SBOM generation deterministic. This is useful for reproducible builds, to allow cleaner inter-diffs while developing new SBOM features. Can be enabled by configuring with -DQT_SBOM_FAKE_DETERMINISTIC_BUILD=ON Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Id0003bdd23fe57abd70213a2a108885861693242 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 10:05:05 +01:00
if(QT_SBOM_FAKE_TIMESTAMP)
set(current_utc "2590-01-01T11:33:55Z")
set(current_year "2590")
else()
string(TIMESTAMP current_utc UTC)
string(TIMESTAMP current_year "%Y" UTC)
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value(PROJECT "${PROJECT_NAME}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Fix sbom git vars not being available in various scopes Initially the git vars were assigned to the parent scope of the _qt_internal_sbom_begin_project function, with the intent to set them in the global scope. But the function was later wrapped in other functions, so the variables stopped being accessible. Instead of playing with recursive PARENT_SCOPEs, save the variables in global properties like we do for other info, and use a new _qt_internal_sbom_get_git_version_vars() function to query the vars in the code that needs them. This fixes generated purls to contain the git version and hashes. Also add a new internal API wrapper macro called qt_internal_sbom_get_git_version_vars to allow calling it in other repos. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I061b34f418c1ecc1c66c8c01ef758d2f40611ede Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 11:46:36 +01:00
_qt_internal_sbom_get_git_version_vars()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(default_sbom_file_name
"${arg_PROJECT}/${arg_PROJECT}-sbom-${QT_SBOM_GIT_VERSION_PATH}.spdx")
set(default_install_sbom_path
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
"\${CMAKE_INSTALL_PREFIX}/${CMAKE_INSTALL_DATAROOTDIR}/${default_sbom_file_name}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value(OUTPUT "${default_install_sbom_path}")
_qt_internal_sbom_set_default_option_value(OUTPUT_RELATIVE_PATH
CMake: Make ninja 'sbom' work for top-level builds We need to pass all dirs where sboms will be generated, for external document referencing. We also need to set up dependencies between the repo sbom custom targets, so that qtsvg sbom is only run after qtbase sbom. We use the dependencies.yaml info to set up the dependencies. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id3331e11742bc2c86e7ed52ce26b3ff21eace359 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-20 19:21:53 +02:00
"${default_sbom_file_name}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value(LICENSE "NOASSERTION")
_qt_internal_sbom_set_default_option_value(PROJECT_FOR_SPDX_ID "Package-${arg_PROJECT}")
_qt_internal_sbom_set_default_option_value_and_error_if_empty(SUPPLIER "")
_qt_internal_sbom_set_default_option_value(COPYRIGHT "${current_year} ${arg_SUPPLIER}")
_qt_internal_sbom_set_default_option_value_and_error_if_empty(SUPPLIER_URL
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
"${PROJECT_HOMEPAGE_URL}")
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value(NAMESPACE
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
"${arg_SUPPLIER}/spdxdocs/${arg_PROJECT}-${QT_SBOM_GIT_VERSION}")
if(arg_CPE)
set(QT_SBOM_CPE "${arg_CPE}")
else()
CMake: Don't add platform CPE to SBOM It triggers vulnerabilities when loading the SBOM into tools that look for CVEs matching those CPEs. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ic4949ca5749ce4e297344760d3500d982c9a4449 Reviewed-by: Kai Köhne <kai.koehne@qt.io>
2024-07-18 16:32:24 +02:00
set(QT_SBOM_CPE "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
string(REGEX REPLACE "[^A-Za-z0-9.]+" "-" arg_PROJECT_FOR_SPDX_ID "${arg_PROJECT_FOR_SPDX_ID}")
string(REGEX REPLACE "-+$" "" arg_PROJECT_FOR_SPDX_ID "${arg_PROJECT_FOR_SPDX_ID}")
# Prevent collision with other generated SPDXID with -[0-9]+ suffix.
string(REGEX REPLACE "-([0-9]+)$" "\\1" arg_PROJECT_FOR_SPDX_ID "${arg_PROJECT_FOR_SPDX_ID}")
set(project_spdx_id "SPDXRef-${arg_PROJECT_FOR_SPDX_ID}")
if(arg_OUT_VAR_PROJECT_SPDX_ID)
set(${arg_OUT_VAR_PROJECT_SPDX_ID} "${project_spdx_id}" PARENT_SCOPE)
endif()
get_filename_component(doc_name "${arg_OUTPUT}" NAME_WLE)
get_cmake_property(is_multi_config GENERATOR_IS_MULTI_CONFIG)
if(is_multi_config)
set(cmake_configs "${CMAKE_CONFIGURATION_TYPES}")
else()
set(cmake_configs "${CMAKE_BUILD_TYPE}")
endif()
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value(DOWNLOAD_LOCATION "NOASSERTION")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Allow adding content to the SBOM repo project comment Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I810814afa69ddf6dc89e834f525b62d87c3aa109 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-17 16:44:31 +02:00
set(cmake_version "Built by CMake ${CMAKE_VERSION}")
set(system_name_and_processor "${CMAKE_SYSTEM_NAME} (${CMAKE_SYSTEM_PROCESSOR})")
set(default_project_comment
"${cmake_version} with ${cmake_configs} configuration for ${system_name_and_processor}")
set(project_comment "${default_project_comment}")
if(arg_PROJECT_COMMENT)
string(APPEND project_comment "${arg_PROJECT_COMMENT}")
endif()
set(project_comment "<text>${project_comment}</text>")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(content
"SPDXVersion: SPDX-2.3
DataLicense: CC0-1.0
SPDXID: SPDXRef-DOCUMENT
DocumentName: ${doc_name}
DocumentNamespace: ${arg_NAMESPACE}
Creator: Organization: ${arg_SUPPLIER}
Creator: Tool: Qt Build System
CreatorComment: <text>This SPDX document was created from CMake ${CMAKE_VERSION}, using the qt
build system from https://code.qt.io/cgit/qt/qtbase.git/tree/cmake/QtPublicSbomHelpers.cmake</text>
Created: ${current_utc}\${QT_SBOM_EXTERNAL_DOC_REFS}
PackageName: ${CMAKE_CXX_COMPILER_ID}
SPDXID: SPDXRef-compiler
PackageVersion: ${CMAKE_CXX_COMPILER_VERSION}
PackageDownloadLocation: NOASSERTION
PackageLicenseConcluded: NOASSERTION
PackageLicenseDeclared: NOASSERTION
PackageCopyrightText: NOASSERTION
PackageSupplier: Organization: Anonymous
FilesAnalyzed: false
PackageSummary: <text>The compiler as identified by CMake, running on ${CMAKE_HOST_SYSTEM_NAME} (${CMAKE_HOST_SYSTEM_PROCESSOR})</text>
PrimaryPackagePurpose: APPLICATION
Relationship: SPDXRef-compiler BUILD_DEPENDENCY_OF ${project_spdx_id}
CMake: Remove the CMAKE_CXX_COMPILER path from the SBOM It causes the SBOM to be non-reproducible, as the path is different on each machine. Embedding just the compiler id and version should be sufficient. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-130557 Change-Id: Iad66ac3f90c4969d27307c77bef46d2eba15c175 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-10-25 16:44:09 +02:00
RelationshipComment: <text>${project_spdx_id} is built by compiler ${CMAKE_CXX_COMPILER_ID} version ${CMAKE_CXX_COMPILER_VERSION}</text>
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
PackageName: ${arg_PROJECT}
SPDXID: ${project_spdx_id}
ExternalRef: SECURITY cpe23Type ${QT_SBOM_CPE}
ExternalRef: PACKAGE-MANAGER purl pkg:generic/${arg_SUPPLIER}/${arg_PROJECT}@${QT_SBOM_GIT_VERSION}
PackageVersion: ${QT_SBOM_GIT_VERSION}
PackageSupplier: Organization: ${arg_SUPPLIER}
PackageDownloadLocation: ${arg_DOWNLOAD_LOCATION}
PackageLicenseConcluded: ${arg_LICENSE}
PackageLicenseDeclared: ${arg_LICENSE}
PackageCopyrightText: ${arg_COPYRIGHT}
PackageHomePage: ${arg_SUPPLIER_URL}
CMake: Allow adding content to the SBOM repo project comment Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I810814afa69ddf6dc89e834f525b62d87c3aa109 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-17 16:44:31 +02:00
PackageComment: ${project_comment}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
PackageVerificationCode: \${QT_SBOM_VERIFICATION_CODE}
BuiltDate: ${current_utc}
Relationship: SPDXRef-DOCUMENT DESCRIBES ${project_spdx_id}
")
# Create the directory that will contain all sbom related files.
_qt_internal_get_current_project_sbom_dir(sbom_dir)
file(MAKE_DIRECTORY "${sbom_dir}")
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
set_property(GLOBAL APPEND PROPERTY _qt_internal_sbom_dirs "${sbom_dir}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Generate project document intro spdx file.
_qt_internal_sbom_get_root_project_name_lower_case(repo_project_name_lowercase)
set(document_intro_file_name
"${sbom_dir}/SPDXRef-DOCUMENT-${repo_project_name_lowercase}.spdx.in")
file(GENERATE OUTPUT "${document_intro_file_name}" CONTENT "${content}")
# This is the file that will be incrementally assembled by having content appended to it.
_qt_internal_get_staging_area_spdx_file_path(staging_area_spdx_file)
CMake: Introduce sbom 'output file path without extensions' variable Some operations might want to install additional files next to the repo sbom, with the same file base name as the repo sbom file name. E.g install qtbase.source.spdx.json next to qtbase.spdx.json. Make the output file path without the extension available in a new QT_SBOM_OUTPUT_PATH_WITHOUT_EXT variable during installation. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0b9442cffa3f3b0fd2387e77569a94e43ce387a5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 18:46:01 +02:00
get_filename_component(output_file_name_without_ext "${arg_OUTPUT}" NAME_WLE)
get_filename_component(output_file_ext "${arg_OUTPUT}" LAST_EXT)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
get_cmake_property(is_multi_config GENERATOR_IS_MULTI_CONFIG)
if(is_multi_config)
set(multi_config_suffix "-$<CONFIG>")
else()
set(multi_config_suffix "")
endif()
CMake: Introduce sbom 'output file path without extensions' variable Some operations might want to install additional files next to the repo sbom, with the same file base name as the repo sbom file name. E.g install qtbase.source.spdx.json next to qtbase.spdx.json. Make the output file path without the extension available in a new QT_SBOM_OUTPUT_PATH_WITHOUT_EXT variable during installation. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0b9442cffa3f3b0fd2387e77569a94e43ce387a5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 18:46:01 +02:00
set(computed_sbom_file_name_without_ext "${output_file_name_without_ext}${multi_config_suffix}")
set(computed_sbom_file_name "${output_file_name_without_ext}${output_file_ext}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Fix location of build time sboms in no-prefix builds For build time sboms to be found in a no-prefix qt build of a repo, specifically when finding external sbom documents, the build time sboms have to be in the same location. Previously they were in each repo's build dir. Align the location to be in the qtbase build dir, just like for super builds. Also check for QT_BUILDING_QT because we don't want to apply the logic for QT_WILL_INSTALL not being set in user projects. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I04a066fc0efc91a4f2ee938c190b8632c8c946aa Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-31 16:05:25 +01:00
# In a super build and in a no-prefix build, put all the build time sboms into the same dir in,
# in the qtbase build dir.
if(QT_BUILDING_QT AND (QT_SUPERBUILD OR (NOT QT_WILL_INSTALL)))
set(build_sbom_root_dir "${QT_BUILD_DIR}")
CMake: Make ninja 'sbom' work for top-level builds We need to pass all dirs where sboms will be generated, for external document referencing. We also need to set up dependencies between the repo sbom custom targets, so that qtsvg sbom is only run after qtbase sbom. We use the dependencies.yaml info to set up the dependencies. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id3331e11742bc2c86e7ed52ce26b3ff21eace359 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-20 19:21:53 +02:00
else()
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
set(build_sbom_root_dir "${sbom_dir}")
CMake: Make ninja 'sbom' work for top-level builds We need to pass all dirs where sboms will be generated, for external document referencing. We also need to set up dependencies between the repo sbom custom targets, so that qtsvg sbom is only run after qtbase sbom. We use the dependencies.yaml info to set up the dependencies. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id3331e11742bc2c86e7ed52ce26b3ff21eace359 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-20 19:21:53 +02:00
endif()
CMake: Introduce sbom 'output file path without extensions' variable Some operations might want to install additional files next to the repo sbom, with the same file base name as the repo sbom file name. E.g install qtbase.source.spdx.json next to qtbase.spdx.json. Make the output file path without the extension available in a new QT_SBOM_OUTPUT_PATH_WITHOUT_EXT variable during installation. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0b9442cffa3f3b0fd2387e77569a94e43ce387a5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 18:46:01 +02:00
get_filename_component(output_relative_dir "${arg_OUTPUT_RELATIVE_PATH}" DIRECTORY)
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
set(build_sbom_dir "${build_sbom_root_dir}/${output_relative_dir}")
set(build_sbom_path "${build_sbom_dir}/${computed_sbom_file_name}")
CMake: Introduce sbom 'output file path without extensions' variable Some operations might want to install additional files next to the repo sbom, with the same file base name as the repo sbom file name. E.g install qtbase.source.spdx.json next to qtbase.spdx.json. Make the output file path without the extension available in a new QT_SBOM_OUTPUT_PATH_WITHOUT_EXT variable during installation. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0b9442cffa3f3b0fd2387e77569a94e43ce387a5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 18:46:01 +02:00
set(build_sbom_path_without_ext
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
"${build_sbom_dir}/${computed_sbom_file_name_without_ext}")
CMake: Introduce sbom 'output file path without extensions' variable Some operations might want to install additional files next to the repo sbom, with the same file base name as the repo sbom file name. E.g install qtbase.source.spdx.json next to qtbase.spdx.json. Make the output file path without the extension available in a new QT_SBOM_OUTPUT_PATH_WITHOUT_EXT variable during installation. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0b9442cffa3f3b0fd2387e77569a94e43ce387a5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 18:46:01 +02:00
set(install_sbom_path "${arg_OUTPUT}")
get_filename_component(install_sbom_dir "${install_sbom_path}" DIRECTORY)
set(install_sbom_path_without_ext "${install_sbom_dir}/${output_file_name_without_ext}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Create cmake file to append the document intro spdx to the staging file.
set(create_staging_file "${sbom_dir}/append_document_to_staging${multi_config_suffix}.cmake")
set(content "
cmake_minimum_required(VERSION 3.16)
message(STATUS \"Starting SBOM generation in build dir: ${staging_area_spdx_file}\")
set(QT_SBOM_EXTERNAL_DOC_REFS \"\")
file(READ \"${document_intro_file_name}\" content)
# Override any previous file because we're starting from scratch.
file(WRITE \"${staging_area_spdx_file}\" \"\${content}\")
")
file(GENERATE OUTPUT "${create_staging_file}" CONTENT "${content}")
CMake: Add a few more internal sbom helper functions Add a few internal functions to allow - getting the sbom project supplier - project supplier url - project namespace - computing a project namespace - computing a project file name - getting a sanitized spdx id given a hint Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I0dc3df274eaf6882a6af021aabee75501b5083f8 Reviewed-by: Moss Heim <moss.heim@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-21 19:32:35 +01:00
set_property(GLOBAL PROPERTY _qt_sbom_project_supplier "${arg_SUPPLIER}")
set_property(GLOBAL PROPERTY _qt_sbom_project_supplier_url "${arg_SUPPLIER_URL}")
set_property(GLOBAL PROPERTY _qt_sbom_project_namespace "${arg_NAMESPACE}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set_property(GLOBAL PROPERTY _qt_sbom_project_name "${arg_PROJECT}")
CMake: Fix project spdx id used in implicit relationships We shouldn't be using the direct project name in spdx relationships, but rather the sanitized name which is prefixed with 'SPDXRef-Package-', to ensure sbom validation succeeds. Also we should default PROJECT_FOR_SPDX_ID to "Package-${arg_PROJECT}" if it's not set. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I354babcf4fea5f6efd9b32422dd8d3835ef50f15 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-08 15:37:41 +01:00
set_property(GLOBAL PROPERTY _qt_sbom_project_spdx_id "${project_spdx_id}")
CMake: Introduce sbom 'output file path without extensions' variable Some operations might want to install additional files next to the repo sbom, with the same file base name as the repo sbom file name. E.g install qtbase.source.spdx.json next to qtbase.spdx.json. Make the output file path without the extension available in a new QT_SBOM_OUTPUT_PATH_WITHOUT_EXT variable during installation. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0b9442cffa3f3b0fd2387e77569a94e43ce387a5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 18:46:01 +02:00
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set_property(GLOBAL PROPERTY _qt_sbom_build_output_path "${build_sbom_path}")
CMake: Introduce sbom 'output file path without extensions' variable Some operations might want to install additional files next to the repo sbom, with the same file base name as the repo sbom file name. E.g install qtbase.source.spdx.json next to qtbase.spdx.json. Make the output file path without the extension available in a new QT_SBOM_OUTPUT_PATH_WITHOUT_EXT variable during installation. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0b9442cffa3f3b0fd2387e77569a94e43ce387a5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 18:46:01 +02:00
set_property(GLOBAL PROPERTY _qt_sbom_build_output_path_without_ext
"${build_sbom_path_without_ext}")
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
set_property(GLOBAL PROPERTY _qt_sbom_build_output_dir "${build_sbom_dir}")
CMake: Introduce sbom 'output file path without extensions' variable Some operations might want to install additional files next to the repo sbom, with the same file base name as the repo sbom file name. E.g install qtbase.source.spdx.json next to qtbase.spdx.json. Make the output file path without the extension available in a new QT_SBOM_OUTPUT_PATH_WITHOUT_EXT variable during installation. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0b9442cffa3f3b0fd2387e77569a94e43ce387a5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 18:46:01 +02:00
set_property(GLOBAL PROPERTY _qt_sbom_install_output_path "${install_sbom_path}")
set_property(GLOBAL PROPERTY _qt_sbom_install_output_path_without_ext
"${install_sbom_path_without_ext}")
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
set_property(GLOBAL PROPERTY _qt_sbom_install_output_dir "${install_sbom_dir}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set_property(GLOBAL APPEND PROPERTY _qt_sbom_cmake_include_files "${create_staging_file}")
set_property(GLOBAL PROPERTY _qt_sbom_spdx_id_count 0)
CMake: Allow adding custom SBOM relationships to targets and projects Add a new SBOM_RELATIONSHIPS option to qt_internal_extend_target and friends that allows adding custom relationships to the current target. Add a new function qt_internal_sbom_add_project_relationship that allows adding custom relationships to the current project SBOM document. A sample usage might be: qt_internal_sbom_get_project_spdx_id(project_spdx_id) qt_internal_sbom_get_target_spdx_id(Svg svg_spdx_id) qt_internal_extend_target(Svg SBOM_RELATIONSHIPS "${svg_spdx_id} DESCENDANT_OF ${project_spdx_id}" ) qt_internal_sbom_add_project_relationship( RELATIONSHIPS "${svg_spdx_id} CONTAINS NOASSERTION" "${svg_spdx_id} DESCRIBES NOASSERTION" "${project_spdx_id} DESCRIBES NOASSERTION" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ie0119ca71b047c7515e1deaf84a5a67ea01b5274 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 19:13:29 +01:00
set_property(GLOBAL PROPERTY _qt_sbom_relationship_counter 0)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endfunction()
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
# Signals the end of recording sbom information for a project.
# Creates an 'sbom' custom target to generate an incomplete sbom at build time (no checksums).
# Creates install rules to install a complete (with checksums) sbom.
function(_qt_internal_sbom_end_project_generate)
get_property(sbom_build_output_path GLOBAL PROPERTY _qt_sbom_build_output_path)
get_property(sbom_build_output_path_without_ext GLOBAL PROPERTY
_qt_sbom_build_output_path_without_ext)
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
get_property(sbom_build_output_dir GLOBAL PROPERTY _qt_sbom_build_output_dir)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
get_property(sbom_install_output_path GLOBAL PROPERTY _qt_sbom_install_output_path)
get_property(sbom_install_output_path_without_ext GLOBAL PROPERTY
_qt_sbom_install_output_path_without_ext)
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
get_property(sbom_install_output_dir GLOBAL PROPERTY _qt_sbom_install_output_dir)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
if(NOT sbom_build_output_path)
message(FATAL_ERROR "Call _qt_internal_sbom_begin_project() first")
endif()
_qt_internal_get_staging_area_spdx_file_path(staging_area_spdx_file)
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
CMake: Add API to include custom cmake files during SBOM generation Allow including custom cmake files at predefined steps of the sbom generation using the new internal qt_internal_sbom_add_cmake_include_step API. This covers pre-existing steps like BEGIN, END, POST_GENERATION, and VERIFY, as well as two new steps, BEFORE_CHECKSUM and AFTER_CHECKSUM. A sample usage of the new API could be: set(step_path "${CMAKE_CURRENT_BINARY_DIR}/run_some_cmake.cmake") file(WRITE "${step_path}" " # This is a CMake script to run some CMake code. message(STATUS \"Running some CMake code...\") message(STATUS \"Done running some CMake code.\") " ) foreach(step IN ITEMS BEGIN END BEFORE_CHECKSUM AFTER_CHECKSUM POST_GENERATION VERIFY) qt_internal_sbom_add_cmake_include_step( STEP "${step}" INCLUDE_PATH "${step_path}" ) endforeach() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I6f9512bf3a3bc3ebeb7d21426e5a3833280e42a5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 17:38:31 +01:00
_qt_internal_sbom_collect_cmake_include_files(includes
JOIN_WITH_NEWLINES
PROPERTIES _qt_sbom_cmake_include_files _qt_sbom_cmake_end_include_files
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Add API to include custom cmake files during SBOM generation Allow including custom cmake files at predefined steps of the sbom generation using the new internal qt_internal_sbom_add_cmake_include_step API. This covers pre-existing steps like BEGIN, END, POST_GENERATION, and VERIFY, as well as two new steps, BEFORE_CHECKSUM and AFTER_CHECKSUM. A sample usage of the new API could be: set(step_path "${CMAKE_CURRENT_BINARY_DIR}/run_some_cmake.cmake") file(WRITE "${step_path}" " # This is a CMake script to run some CMake code. message(STATUS \"Running some CMake code...\") message(STATUS \"Done running some CMake code.\") " ) foreach(step IN ITEMS BEGIN END BEFORE_CHECKSUM AFTER_CHECKSUM POST_GENERATION VERIFY) qt_internal_sbom_add_cmake_include_step( STEP "${step}" INCLUDE_PATH "${step_path}" ) endforeach() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I6f9512bf3a3bc3ebeb7d21426e5a3833280e42a5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 17:38:31 +01:00
# Before checksum includes are included after the verification codes have been collected
# and before their merged checksum(s) has been computed.
_qt_internal_sbom_collect_cmake_include_files(before_checksum_includes
JOIN_WITH_NEWLINES
PROPERTIES _qt_sbom_cmake_before_checksum_include_files
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Add API to include custom cmake files during SBOM generation Allow including custom cmake files at predefined steps of the sbom generation using the new internal qt_internal_sbom_add_cmake_include_step API. This covers pre-existing steps like BEGIN, END, POST_GENERATION, and VERIFY, as well as two new steps, BEFORE_CHECKSUM and AFTER_CHECKSUM. A sample usage of the new API could be: set(step_path "${CMAKE_CURRENT_BINARY_DIR}/run_some_cmake.cmake") file(WRITE "${step_path}" " # This is a CMake script to run some CMake code. message(STATUS \"Running some CMake code...\") message(STATUS \"Done running some CMake code.\") " ) foreach(step IN ITEMS BEGIN END BEFORE_CHECKSUM AFTER_CHECKSUM POST_GENERATION VERIFY) qt_internal_sbom_add_cmake_include_step( STEP "${step}" INCLUDE_PATH "${step_path}" ) endforeach() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I6f9512bf3a3bc3ebeb7d21426e5a3833280e42a5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 17:38:31 +01:00
# After checksum includes are included after the checksum has been computed and written to the
# QT_SBOM_VERIFICATION_CODE variable.
_qt_internal_sbom_collect_cmake_include_files(after_checksum_includes
JOIN_WITH_NEWLINES
PROPERTIES _qt_sbom_cmake_after_checksum_include_files
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
# Post generation includes are included for both build and install time sboms, after
# sbom generation has finished.
CMake: Add API to include custom cmake files during SBOM generation Allow including custom cmake files at predefined steps of the sbom generation using the new internal qt_internal_sbom_add_cmake_include_step API. This covers pre-existing steps like BEGIN, END, POST_GENERATION, and VERIFY, as well as two new steps, BEFORE_CHECKSUM and AFTER_CHECKSUM. A sample usage of the new API could be: set(step_path "${CMAKE_CURRENT_BINARY_DIR}/run_some_cmake.cmake") file(WRITE "${step_path}" " # This is a CMake script to run some CMake code. message(STATUS \"Running some CMake code...\") message(STATUS \"Done running some CMake code.\") " ) foreach(step IN ITEMS BEGIN END BEFORE_CHECKSUM AFTER_CHECKSUM POST_GENERATION VERIFY) qt_internal_sbom_add_cmake_include_step( STEP "${step}" INCLUDE_PATH "${step_path}" ) endforeach() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I6f9512bf3a3bc3ebeb7d21426e5a3833280e42a5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 17:38:31 +01:00
_qt_internal_sbom_collect_cmake_include_files(post_generation_includes
JOIN_WITH_NEWLINES
PROPERTIES _qt_sbom_cmake_post_generation_include_files
)
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Verification only makes sense on installation, where the checksums are present.
CMake: Add API to include custom cmake files during SBOM generation Allow including custom cmake files at predefined steps of the sbom generation using the new internal qt_internal_sbom_add_cmake_include_step API. This covers pre-existing steps like BEGIN, END, POST_GENERATION, and VERIFY, as well as two new steps, BEFORE_CHECKSUM and AFTER_CHECKSUM. A sample usage of the new API could be: set(step_path "${CMAKE_CURRENT_BINARY_DIR}/run_some_cmake.cmake") file(WRITE "${step_path}" " # This is a CMake script to run some CMake code. message(STATUS \"Running some CMake code...\") message(STATUS \"Done running some CMake code.\") " ) foreach(step IN ITEMS BEGIN END BEFORE_CHECKSUM AFTER_CHECKSUM POST_GENERATION VERIFY) qt_internal_sbom_add_cmake_include_step( STEP "${step}" INCLUDE_PATH "${step_path}" ) endforeach() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I6f9512bf3a3bc3ebeb7d21426e5a3833280e42a5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 17:38:31 +01:00
_qt_internal_sbom_collect_cmake_include_files(verify_includes
JOIN_WITH_NEWLINES
PROPERTIES _qt_sbom_cmake_verify_include_files
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
get_cmake_property(is_multi_config GENERATOR_IS_MULTI_CONFIG)
if(is_multi_config)
set(multi_config_suffix "-$<CONFIG>")
else()
set(multi_config_suffix "")
endif()
_qt_internal_get_current_project_sbom_dir(sbom_dir)
set(content "
# QT_SBOM_BUILD_TIME be set to FALSE at install time, so don't override if it's set.
# This allows reusing the same cmake file for both build and install.
if(NOT DEFINED QT_SBOM_BUILD_TIME)
set(QT_SBOM_BUILD_TIME TRUE)
endif()
if(NOT QT_SBOM_OUTPUT_PATH)
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
set(QT_SBOM_OUTPUT_DIR \"${sbom_build_output_dir}\")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(QT_SBOM_OUTPUT_PATH \"${sbom_build_output_path}\")
CMake: Introduce sbom 'output file path without extensions' variable Some operations might want to install additional files next to the repo sbom, with the same file base name as the repo sbom file name. E.g install qtbase.source.spdx.json next to qtbase.spdx.json. Make the output file path without the extension available in a new QT_SBOM_OUTPUT_PATH_WITHOUT_EXT variable during installation. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0b9442cffa3f3b0fd2387e77569a94e43ce387a5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 18:46:01 +02:00
set(QT_SBOM_OUTPUT_PATH_WITHOUT_EXT \"${sbom_build_output_path_without_ext}\")
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
file(MAKE_DIRECTORY \"${sbom_build_output_dir}\")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
set(QT_SBOM_VERIFICATION_CODES \"\")
${includes}
if(QT_SBOM_BUILD_TIME)
message(STATUS \"Finalizing SBOM generation in build dir: \${QT_SBOM_OUTPUT_PATH}\")
configure_file(\"${staging_area_spdx_file}\" \"\${QT_SBOM_OUTPUT_PATH}\")
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
${post_generation_includes}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
")
set(assemble_sbom "${sbom_dir}/assemble_sbom${multi_config_suffix}.cmake")
file(GENERATE OUTPUT "${assemble_sbom}" CONTENT "${content}")
if(NOT TARGET sbom)
add_custom_target(sbom)
endif()
_qt_internal_sbom_get_root_project_name_lower_case(repo_project_name_lowercase)
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
_qt_internal_sbom_get_qt_repo_project_name_lower_case(real_qt_repo_project_name_lowercase)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Create a build target to create a build-time sbom (no verification codes or sha1s).
set(repo_sbom_target "sbom_${repo_project_name_lowercase}")
set(comment "")
string(APPEND comment "Assembling build time SPDX document without checksums for "
"${repo_project_name_lowercase}. Just for testing.")
add_custom_target(${repo_sbom_target}
COMMAND "${CMAKE_COMMAND}" -P "${assemble_sbom}"
COMMENT "${comment}"
VERBATIM
USES_TERMINAL # To avoid running two configs of the command in parallel
)
CMake: Make ninja 'sbom' work for top-level builds We need to pass all dirs where sboms will be generated, for external document referencing. We also need to set up dependencies between the repo sbom custom targets, so that qtsvg sbom is only run after qtbase sbom. We use the dependencies.yaml info to set up the dependencies. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id3331e11742bc2c86e7ed52ce26b3ff21eace359 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-20 19:21:53 +02:00
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
get_cmake_property(qt_repo_deps _qt_repo_deps_${real_qt_repo_project_name_lowercase})
CMake: Make ninja 'sbom' work for top-level builds We need to pass all dirs where sboms will be generated, for external document referencing. We also need to set up dependencies between the repo sbom custom targets, so that qtsvg sbom is only run after qtbase sbom. We use the dependencies.yaml info to set up the dependencies. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id3331e11742bc2c86e7ed52ce26b3ff21eace359 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-20 19:21:53 +02:00
if(qt_repo_deps)
foreach(repo_dep IN LISTS qt_repo_deps)
set(repo_dep_sbom "sbom_${repo_dep}")
if(TARGET "${repo_dep_sbom}")
add_dependencies(${repo_sbom_target} ${repo_dep_sbom})
endif()
endforeach()
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
add_dependencies(sbom ${repo_sbom_target})
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
# Add 'reuse lint' per-repo custom targets.
if(arg_LINT_SOURCE_SBOM AND NOT QT_INTERNAL_NO_SBOM_PYTHON_OPS)
if(NOT TARGET reuse_lint)
add_custom_target(reuse_lint)
endif()
set(comment "Running 'reuse lint' for '${repo_project_name_lowercase}'.")
add_custom_target(${repo_sbom_target}_reuse_lint
COMMAND "${CMAKE_COMMAND}" -P "${reuse_lint_script}"
COMMENT "${comment}"
VERBATIM
USES_TERMINAL # To avoid running multiple lints in parallel
)
add_dependencies(reuse_lint ${repo_sbom_target}_reuse_lint)
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(extra_code_begin "")
set(extra_code_inner_end "")
get_cmake_property(is_multi_config GENERATOR_IS_MULTI_CONFIG)
if(is_multi_config)
set(configs ${CMAKE_CONFIGURATION_TYPES})
set(install_markers_dir "${sbom_dir}")
set(install_marker_path "${install_markers_dir}/finished_install-$<CONFIG>.cmake")
set(install_marker_code "
message(STATUS \"Writing install marker for config $<CONFIG>: ${install_marker_path} \")
file(WRITE \"${install_marker_path}\" \"\")
")
install(CODE "${install_marker_code}" COMPONENT sbom)
if(QT_SUPERBUILD)
install(CODE "${install_marker_code}" COMPONENT "sbom_${repo_project_name_lowercase}"
EXCLUDE_FROM_ALL)
endif()
set(install_markers "")
foreach(config IN LISTS configs)
set(marker_path "${install_markers_dir}/finished_install-${config}.cmake")
list(APPEND install_markers "${marker_path}")
# Remove the markers on reconfiguration, just in case there are stale ones.
if(EXISTS "${marker_path}")
file(REMOVE "${marker_path}")
endif()
endforeach()
set(extra_code_begin "
set(QT_SBOM_INSTALL_MARKERS \"${install_markers}\")
foreach(QT_SBOM_INSTALL_MARKER IN LISTS QT_SBOM_INSTALL_MARKERS)
if(NOT EXISTS \"\${QT_SBOM_INSTALL_MARKER}\")
set(QT_SBOM_INSTALLED_ALL_CONFIGS FALSE)
endif()
endforeach()
")
set(extra_code_inner_end "
foreach(QT_SBOM_INSTALL_MARKER IN LISTS QT_SBOM_INSTALL_MARKERS)
message(STATUS
\"Removing install marker: \${QT_SBOM_INSTALL_MARKER} \")
file(REMOVE \"\${QT_SBOM_INSTALL_MARKER}\")
endforeach()
")
endif()
# Allow skipping checksum computation for testing purposes, while installing just the sbom
# documents, without requiring to build and install all the actual files.
CMake: Add a fake deterministic SBOM generation option Replaces content like timestamps, version strings and checksums with constant values to make the SBOM generation deterministic. This is useful for reproducible builds, to allow cleaner inter-diffs while developing new SBOM features. Can be enabled by configuring with -DQT_SBOM_FAKE_DETERMINISTIC_BUILD=ON Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Id0003bdd23fe57abd70213a2a108885861693242 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 10:05:05 +01:00
if(QT_SBOM_FAKE_CHECKSUM)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
string(APPEND extra_code_begin "
set(QT_SBOM_FAKE_CHECKSUM TRUE)")
endif()
set(assemble_sbom_install "
set(QT_SBOM_INSTALLED_ALL_CONFIGS TRUE)
${extra_code_begin}
if(QT_SBOM_INSTALLED_ALL_CONFIGS)
set(QT_SBOM_BUILD_TIME FALSE)
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
set(QT_SBOM_OUTPUT_DIR \"${sbom_install_output_dir}\")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(QT_SBOM_OUTPUT_PATH \"${sbom_install_output_path}\")
CMake: Introduce sbom 'output file path without extensions' variable Some operations might want to install additional files next to the repo sbom, with the same file base name as the repo sbom file name. E.g install qtbase.source.spdx.json next to qtbase.spdx.json. Make the output file path without the extension available in a new QT_SBOM_OUTPUT_PATH_WITHOUT_EXT variable during installation. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0b9442cffa3f3b0fd2387e77569a94e43ce387a5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 18:46:01 +02:00
set(QT_SBOM_OUTPUT_PATH_WITHOUT_EXT \"${sbom_install_output_path_without_ext}\")
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
file(MAKE_DIRECTORY \"${sbom_install_output_dir}\")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
include(\"${assemble_sbom}\")
CMake: Add API to include custom cmake files during SBOM generation Allow including custom cmake files at predefined steps of the sbom generation using the new internal qt_internal_sbom_add_cmake_include_step API. This covers pre-existing steps like BEGIN, END, POST_GENERATION, and VERIFY, as well as two new steps, BEFORE_CHECKSUM and AFTER_CHECKSUM. A sample usage of the new API could be: set(step_path "${CMAKE_CURRENT_BINARY_DIR}/run_some_cmake.cmake") file(WRITE "${step_path}" " # This is a CMake script to run some CMake code. message(STATUS \"Running some CMake code...\") message(STATUS \"Done running some CMake code.\") " ) foreach(step IN ITEMS BEGIN END BEFORE_CHECKSUM AFTER_CHECKSUM POST_GENERATION VERIFY) qt_internal_sbom_add_cmake_include_step( STEP "${step}" INCLUDE_PATH "${step_path}" ) endforeach() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I6f9512bf3a3bc3ebeb7d21426e5a3833280e42a5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 17:38:31 +01:00
${before_checksum_includes}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
list(SORT QT_SBOM_VERIFICATION_CODES)
string(REPLACE \";\" \"\" QT_SBOM_VERIFICATION_CODES \"\${QT_SBOM_VERIFICATION_CODES}\")
file(WRITE \"${sbom_dir}/verification.txt\" \"\${QT_SBOM_VERIFICATION_CODES}\")
file(SHA1 \"${sbom_dir}/verification.txt\" QT_SBOM_VERIFICATION_CODE)
CMake: Add API to include custom cmake files during SBOM generation Allow including custom cmake files at predefined steps of the sbom generation using the new internal qt_internal_sbom_add_cmake_include_step API. This covers pre-existing steps like BEGIN, END, POST_GENERATION, and VERIFY, as well as two new steps, BEFORE_CHECKSUM and AFTER_CHECKSUM. A sample usage of the new API could be: set(step_path "${CMAKE_CURRENT_BINARY_DIR}/run_some_cmake.cmake") file(WRITE "${step_path}" " # This is a CMake script to run some CMake code. message(STATUS \"Running some CMake code...\") message(STATUS \"Done running some CMake code.\") " ) foreach(step IN ITEMS BEGIN END BEFORE_CHECKSUM AFTER_CHECKSUM POST_GENERATION VERIFY) qt_internal_sbom_add_cmake_include_step( STEP "${step}" INCLUDE_PATH "${step_path}" ) endforeach() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I6f9512bf3a3bc3ebeb7d21426e5a3833280e42a5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 17:38:31 +01:00
${after_checksum_includes}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
message(STATUS \"Finalizing SBOM generation in install dir: \${QT_SBOM_OUTPUT_PATH}\")
configure_file(\"${staging_area_spdx_file}\" \"\${QT_SBOM_OUTPUT_PATH}\")
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
${post_generation_includes}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
${verify_includes}
${extra_code_inner_end}
else()
message(STATUS \"Skipping SBOM finalization because not all configs were installed.\")
endif()
")
install(CODE "${assemble_sbom_install}" COMPONENT sbom)
if(QT_SUPERBUILD)
install(CODE "${assemble_sbom_install}" COMPONENT "sbom_${repo_project_name_lowercase}"
EXCLUDE_FROM_ALL)
endif()
# Clean up properties, so that they are empty for possible next repo in a top-level build.
set_property(GLOBAL PROPERTY _qt_sbom_cmake_include_files "")
set_property(GLOBAL PROPERTY _qt_sbom_cmake_end_include_files "")
CMake: Add API to include custom cmake files during SBOM generation Allow including custom cmake files at predefined steps of the sbom generation using the new internal qt_internal_sbom_add_cmake_include_step API. This covers pre-existing steps like BEGIN, END, POST_GENERATION, and VERIFY, as well as two new steps, BEFORE_CHECKSUM and AFTER_CHECKSUM. A sample usage of the new API could be: set(step_path "${CMAKE_CURRENT_BINARY_DIR}/run_some_cmake.cmake") file(WRITE "${step_path}" " # This is a CMake script to run some CMake code. message(STATUS \"Running some CMake code...\") message(STATUS \"Done running some CMake code.\") " ) foreach(step IN ITEMS BEGIN END BEFORE_CHECKSUM AFTER_CHECKSUM POST_GENERATION VERIFY) qt_internal_sbom_add_cmake_include_step( STEP "${step}" INCLUDE_PATH "${step_path}" ) endforeach() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I6f9512bf3a3bc3ebeb7d21426e5a3833280e42a5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 17:38:31 +01:00
set_property(GLOBAL PROPERTY _qt_sbom_cmake_before_checksum_include_files "")
set_property(GLOBAL PROPERTY _qt_sbom_cmake_after_checksum_include_files "")
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
set_property(GLOBAL PROPERTY _qt_sbom_cmake_post_generation_include_files "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set_property(GLOBAL PROPERTY _qt_sbom_cmake_verify_include_files "")
endfunction()
CMake: Add API to include custom cmake files during SBOM generation Allow including custom cmake files at predefined steps of the sbom generation using the new internal qt_internal_sbom_add_cmake_include_step API. This covers pre-existing steps like BEGIN, END, POST_GENERATION, and VERIFY, as well as two new steps, BEFORE_CHECKSUM and AFTER_CHECKSUM. A sample usage of the new API could be: set(step_path "${CMAKE_CURRENT_BINARY_DIR}/run_some_cmake.cmake") file(WRITE "${step_path}" " # This is a CMake script to run some CMake code. message(STATUS \"Running some CMake code...\") message(STATUS \"Done running some CMake code.\") " ) foreach(step IN ITEMS BEGIN END BEFORE_CHECKSUM AFTER_CHECKSUM POST_GENERATION VERIFY) qt_internal_sbom_add_cmake_include_step( STEP "${step}" INCLUDE_PATH "${step_path}" ) endforeach() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I6f9512bf3a3bc3ebeb7d21426e5a3833280e42a5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 17:38:31 +01:00
# Gets a list of cmake include file paths, joins them as include() statements and returns the
# output.
function(_qt_internal_sbom_collect_cmake_include_files out_var)
set(opt_args
JOIN_WITH_NEWLINES
)
set(single_args "")
set(multi_args
PROPERTIES
)
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
if(NOT arg_PROPERTIES)
message(FATAL_ERROR "PROPERTIES is required")
endif()
set(includes "")
foreach(property IN LISTS arg_PROPERTIES)
get_cmake_property(include_files "${property}")
if(include_files)
foreach(include_file IN LISTS include_files)
list(APPEND includes "include(\"${include_file}\")")
endforeach()
endif()
endforeach()
if(arg_JOIN_WITH_NEWLINES)
list(JOIN includes "\n" includes)
endif()
set(${out_var} "${includes}" PARENT_SCOPE)
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Helper to add info about a file to the sbom.
# Targets are backed by multiple files in multi-config builds. To support multi-config,
# we generate a -$<CONFIG> file for each config, but we only include / install the one that is
# specified via the CONFIG option.
# For build time sboms, we skip checking file existence and sha1 computation, because the files
# are not installed yet.
function(_qt_internal_sbom_generate_add_file)
set(opt_args
OPTIONAL
)
set(single_args
FILENAME
FILETYPE
RELATIONSHIP
SPDXID
CONFIG
LICENSE
COPYRIGHT
INSTALL_PREFIX
)
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value_and_error_if_empty(FILENAME "")
_qt_internal_sbom_set_default_option_value_and_error_if_empty(FILETYPE "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
set(check_option "")
if(arg_SPDXID)
set(check_option "CHECK" "${arg_SPDXID}")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_get_and_check_spdx_id(
VARIABLE arg_SPDXID
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
${check_option}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
HINTS "SPDXRef-${arg_FILENAME}"
)
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value(LICENSE "NOASSERTION")
_qt_internal_sbom_set_default_option_value(COPYRIGHT "NOASSERTION")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Fix project spdx id used in implicit relationships We shouldn't be using the direct project name in spdx relationships, but rather the sanitized name which is prefixed with 'SPDXRef-Package-', to ensure sbom validation succeeds. Also we should default PROJECT_FOR_SPDX_ID to "Package-${arg_PROJECT}" if it's not set. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I354babcf4fea5f6efd9b32422dd8d3835ef50f15 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-08 15:37:41 +01:00
get_property(sbom_project_spdx_id GLOBAL PROPERTY _qt_sbom_project_spdx_id)
if(NOT sbom_project_spdx_id)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
message(FATAL_ERROR "Call _qt_internal_sbom_begin_project() first")
endif()
if(NOT arg_RELATIONSHIP)
CMake: Fix project spdx id used in implicit relationships We shouldn't be using the direct project name in spdx relationships, but rather the sanitized name which is prefixed with 'SPDXRef-Package-', to ensure sbom validation succeeds. Also we should default PROJECT_FOR_SPDX_ID to "Package-${arg_PROJECT}" if it's not set. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I354babcf4fea5f6efd9b32422dd8d3835ef50f15 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-08 15:37:41 +01:00
set(arg_RELATIONSHIP "${sbom_project_spdx_id} CONTAINS ${arg_SPDXID}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
else()
string(REPLACE
"@QT_SBOM_LAST_SPDXID@" "${arg_SPDXID}" arg_RELATIONSHIP "${arg_RELATIONSHIP}")
endif()
set(fields "")
if(arg_LICENSE)
set(fields "${fields}
LicenseConcluded: ${arg_LICENSE}"
)
else()
set(fields "${fields}
LicenseConcluded: NOASSERTION"
)
endif()
if(arg_COPYRIGHT)
set(fields "${fields}
FileCopyrightText: ${arg_COPYRIGHT}"
)
else()
set(fields "${fields}
FileCopyrightText: NOASSERTION"
)
endif()
set(file_suffix_to_generate "")
get_cmake_property(is_multi_config GENERATOR_IS_MULTI_CONFIG)
if(is_multi_config)
set(file_suffix_to_generate "-$<CONFIG>")
endif()
if(arg_CONFIG)
set(file_suffix_to_install "-${arg_CONFIG}")
else()
set(file_suffix_to_install "")
endif()
_qt_internal_get_staging_area_spdx_file_path(staging_area_spdx_file)
if(arg_INSTALL_PREFIX)
set(install_prefix "${arg_INSTALL_PREFIX}")
else()
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
# The variable is escaped, so it is evaluated during cmake install time, so that the value
# can be overridden with cmake --install . --prefix <path>.
set(install_prefix "\${CMAKE_INSTALL_PREFIX}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
set(content "
CMake: Fix file path in checksum computation to allow spaces Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ica5f830e6d7ca9d8acbc13ebec543ee3cf96b3fe Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-09 14:27:21 +01:00
if(NOT EXISTS \"\$ENV{DESTDIR}${install_prefix}/${arg_FILENAME}\"
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
AND NOT QT_SBOM_BUILD_TIME AND NOT QT_SBOM_FAKE_CHECKSUM)
if(NOT ${arg_OPTIONAL})
CMake: Fix DESTDIR handling in SBOM checksum verification The DESTDIR env var needs to be evaluated at install time, not configure time, so the $ENV{} needs to be escaped in the generated script. Also improve the error message a bit, to say which path exactly it expected to read. This fixes SBOM generation for the debian package builds. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6869f0054601dcb35ae7ad7b55f808df4dc12311 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-17 20:40:18 +02:00
message(FATAL_ERROR \"Cannot find '${arg_FILENAME}' to compute its checksum. \"
\"Expected to find it at '\$ENV{DESTDIR}${install_prefix}/${arg_FILENAME}' \")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
else()
if(NOT QT_SBOM_BUILD_TIME)
if(QT_SBOM_FAKE_CHECKSUM)
set(sha1 \"158942a783ee1095eafacaffd93de73edeadbeef\")
else()
CMake: Fix file path in checksum computation to allow spaces Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ica5f830e6d7ca9d8acbc13ebec543ee3cf96b3fe Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-09 14:27:21 +01:00
file(SHA1 \"\$ENV{DESTDIR}${install_prefix}/${arg_FILENAME}\" sha1)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
list(APPEND QT_SBOM_VERIFICATION_CODES \${sha1})
endif()
file(APPEND \"${staging_area_spdx_file}\"
\"
FileName: ./${arg_FILENAME}
SPDXID: ${arg_SPDXID}
FileType: ${arg_FILETYPE}
FileChecksum: SHA1: \${sha1}${fields}
LicenseInfoInFile: NOASSERTION
Relationship: ${arg_RELATIONSHIP}
\"
)
endif()
")
_qt_internal_get_current_project_sbom_dir(sbom_dir)
set(file_sbom "${sbom_dir}/${arg_SPDXID}${file_suffix_to_generate}.cmake")
file(GENERATE OUTPUT "${file_sbom}" CONTENT "${content}")
set(file_sbom_to_install "${sbom_dir}/${arg_SPDXID}${file_suffix_to_install}.cmake")
set_property(GLOBAL APPEND PROPERTY _qt_sbom_cmake_include_files "${file_sbom_to_install}")
endfunction()
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
# Helper to add a reference to an external SPDX document.
#
# EXTERNAL_DOCUMENT_SPDX_ID: The spdx id by which the external document should be referenced in
# the current project SPDX document. This semantically serves as a pointer to the external document
# URI.
# e.g. DocumentRef-qtbase.
#
# EXTERNAL_DOCUMENT_FILE_PATH: The relative file path of the external sbom document.
# e.g. "sbom/qtbase-6.9.0.spdx"
#
# Can contain generator expressions.
# The file path is searched for in the directories specified by the
# EXTERNAL_DOCUMENT_INSTALL_PREFIXES option during sbom generation, to compute the file checksum.
# The file path is NOT embedded into the current project spdx document.
# Only its spdx id and namespace is embedded. The namespace is extracted from the contents of the
# referenced file.
#
# EXTERNAL_DOCUMENT_INSTALL_PREFIXES: A list of directories where the external document file path
# is searched for. The first existing file is used. Additionally the following locations are
# searched:
# - QT6_INSTALL_PREFIX
# - QT_ADDITIONAL_PACKAGES_PREFIX_PATH
# - QT_ADDITIONAL_SBOM_DOCUMENT_PATHS
#
# EXTERNAL_PACKAGE_SPDX_ID: If set, and no RELATIONSHIP_STRING is provided, an automatic DEPENDS_ON
# relationship is added from the current project spdx id to the package identified by
# $EXTERNAL_DOCUMENT_SPDX_ID:$EXTERNAL_PACKAGE_SPDX_ID options.
# This is mostly a beginner convenience.
#
# RELATIONSHIP_STRING: If set, it is used as the relationship string to add to the current project
# relationships.
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
function(_qt_internal_sbom_generate_add_external_reference)
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
if(NOT QT_GENERATE_SBOM)
return()
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
set(opt_args "")
set(single_args
EXTERNAL_DOCUMENT_FILE_PATH
EXTERNAL_DOCUMENT_SPDX_ID
EXTERNAL_PACKAGE_SPDX_ID
RELATIONSHIP_STRING
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
set(multi_args
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
EXTERNAL_DOCUMENT_INSTALL_PREFIXES
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value_and_error_if_empty(EXTERNAL_DOCUMENT_FILE_PATH "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
if(NOT arg_EXTERNAL_DOCUMENT_SPDX_ID)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
get_property(spdx_id_count GLOBAL PROPERTY _qt_sbom_spdx_id_count)
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
set(arg_EXTERNAL_DOCUMENT_SPDX_ID "DocumentRef-${spdx_id_count}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
math(EXPR spdx_id_count "${spdx_id_count} + 1")
set_property(GLOBAL PROPERTY _qt_sbom_spdx_id_count "${spdx_id_count}")
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
elseif(NOT "${arg_EXTERNAL_DOCUMENT_SPDX_ID}" MATCHES
"^DocumentRef-[a-zA-Z0-9]+[-a-zA-Z0-9]+$")
message(FATAL_ERROR "Invalid DocumentRef \"${arg_EXTERNAL_DOCUMENT_SPDX_ID}\"")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Fix project spdx id used in implicit relationships We shouldn't be using the direct project name in spdx relationships, but rather the sanitized name which is prefixed with 'SPDXRef-Package-', to ensure sbom validation succeeds. Also we should default PROJECT_FOR_SPDX_ID to "Package-${arg_PROJECT}" if it's not set. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I354babcf4fea5f6efd9b32422dd8d3835ef50f15 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-08 15:37:41 +01:00
get_property(sbom_project_spdx_id GLOBAL PROPERTY _qt_sbom_project_spdx_id)
if(NOT sbom_project_spdx_id)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
message(FATAL_ERROR "Call _qt_internal_sbom_begin_project() first")
endif()
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
if(arg_RELATIONSHIP_STRING STREQUAL "")
if(arg_EXTERNAL_PACKAGE_SPDX_ID)
set(external_package "${arg_EXTERNAL_DOCUMENT_SPDX_ID}:${arg_EXTERNAL_PACKAGE_SPDX_ID}")
set(arg_RELATIONSHIP_STRING
CMake: Fix project spdx id used in implicit relationships We shouldn't be using the direct project name in spdx relationships, but rather the sanitized name which is prefixed with 'SPDXRef-Package-', to ensure sbom validation succeeds. Also we should default PROJECT_FOR_SPDX_ID to "Package-${arg_PROJECT}" if it's not set. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I354babcf4fea5f6efd9b32422dd8d3835ef50f15 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-08 15:37:41 +01:00
"${sbom_project_spdx_id} DEPENDS_ON ${external_package}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
else()
string(REPLACE
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
"@QT_SBOM_LAST_SPDXID@" "${arg_EXTERNAL_DOCUMENT_SPDX_ID}"
arg_RELATIONSHIP_STRING "${arg_RELATIONSHIP_STRING}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
_qt_internal_get_staging_area_spdx_file_path(staging_area_spdx_file)
set(install_prefixes "")
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
CMake: Add way to convert JSON SPDX to tag/value SPDX document Add a new qt_internal_sbom_generate_tag_value_spdx_document function that takes an input SPDX JSON file and generates a tag/value SPDX file. This is needed by WebEngine to convert the Chromium JSON file to a tag/value SPDX file so we can reference it as an external document. To ensure the external document is found, we now always add the current sbom build directories as install prefixes. This was previously done only for top-level builds. To ensure the converted external document is referenced only after it is converted, it needs to be converted before any targets mention packages from within it. A sample usage might be: qt_internal_sbom_generate_tag_value_spdx_document( OPERATION_ID qt5compat INPUT_JSON_FILE_PATH "${external_sbom_file_path}" OUT_VAR_OUTPUT_FILE_NAME external_output_file_name OUT_VAR_OUTPUT_ABSOLUTE_FILE_PATH external_output_file_path ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I5d5397f788c8c7960b6fc233c2868244e5816e0b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 20:01:12 +01:00
# Add the current sbom build dirs as install prefixes, so that we can use ninja 'sbom'
# in top-level builds. This is needed because the external references will point
# to sbom docs in different build dirs, not just one.
# We also need it in case we are converting a json document to a tag/value format in the
# current build dir of the project, and want it to be found.
get_cmake_property(build_sbom_dirs _qt_internal_sbom_dirs)
if(build_sbom_dirs)
foreach(build_sbom_dir IN LISTS build_sbom_dirs)
list(APPEND install_prefixes "${build_sbom_dir}")
endforeach()
endif()
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
# Always append the install time install prefix.
# The variable is escaped, so it is evaluated during cmake install time, so that the value
# can be overridden with cmake --install . --prefix <path>.
CMake: Handle DESTDIR for SBOM doc references in top-level builds When installing a top-level build using the DESTDIR env var mechanism, the SBOM generation process could not find the qtbase sbom document, while generating the qtshadertools document. Make sure to prepend the DESTDIR env var to the install-time install prefix, when looking for external SBOM documents. Pick-to: 6.8 6.9 Fixes: QTBUG-132188 Change-Id: I1cdb75842fc552b43d982f05444b5ddc1fe58595 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-03 16:32:42 +01:00
list(APPEND install_prefixes "\$ENV{DESTDIR}\${CMAKE_INSTALL_PREFIX}")
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
if(arg_EXTERNAL_DOCUMENT_INSTALL_PREFIXES)
list(APPEND install_prefixes ${arg_EXTERNAL_DOCUMENT_INSTALL_PREFIXES})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(QT6_INSTALL_PREFIX)
list(APPEND install_prefixes ${QT6_INSTALL_PREFIX})
endif()
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(QT_ADDITIONAL_PACKAGES_PREFIX_PATH)
list(APPEND install_prefixes ${QT_ADDITIONAL_PACKAGES_PREFIX_PATH})
endif()
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(QT_ADDITIONAL_SBOM_DOCUMENT_PATHS)
list(APPEND install_prefixes ${QT_ADDITIONAL_SBOM_DOCUMENT_PATHS})
endif()
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
list(REMOVE_DUPLICATES install_prefixes)
set(relationship_content "")
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
if(arg_RELATIONSHIP_STRING)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(relationship_content "
file(APPEND \"${staging_area_spdx_file}\"
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
\"Relationship: ${arg_RELATIONSHIP_STRING}\")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
")
endif()
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
# File path may not exist yet, and it could be a generator expression.
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(content "
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
set(relative_file_name \"${arg_EXTERNAL_DOCUMENT_FILE_PATH}\")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(document_dir_paths ${install_prefixes})
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
list(JOIN document_dir_paths \"\\n\" document_dir_paths_per_line)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
foreach(document_dir_path IN LISTS document_dir_paths)
set(document_file_path \"\${document_dir_path}/\${relative_file_name}\")
if(EXISTS \"\${document_file_path}\")
break()
endif()
endforeach()
if(NOT EXISTS \"\${document_file_path}\")
message(FATAL_ERROR \"Could not find external SBOM document \${relative_file_name}\"
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
\" in any of the document dir paths: \${document_dir_paths_per_line} \"
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
endif()
file(SHA1 \"\${document_file_path}\" ext_sha1)
file(READ \"\${document_file_path}\" ext_content)
if(NOT \"\${ext_content}\" MATCHES \"[\\r\\n]DocumentNamespace:\")
message(FATAL_ERROR \"Missing DocumentNamespace in \${document_file_path}\")
endif()
string(REGEX REPLACE \"^.*[\\r\\n]DocumentNamespace:[ \\t]*([^#\\r\\n]*).*$\"
\"\\\\1\" ext_ns \"\${ext_content}\")
CMake: Remove semicolon from SBOM external document references The external doc references need to be appended as strings, not list elements, otherwise there will be semicolons in the output SPDX file. This issue was inherited as-is from the upstream SBOM generation project. We are fortunate the json converted files are missing the semicolons due to implementation specifics. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I2b91775d59d64709b72ff7130dd4f6b506ef8244 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-03-03 12:50:15 +01:00
string(APPEND QT_SBOM_EXTERNAL_DOC_REFS \"
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
ExternalDocumentRef: ${arg_EXTERNAL_DOCUMENT_SPDX_ID} \${ext_ns} SHA1: \${ext_sha1}\")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
${relationship_content}
")
_qt_internal_get_current_project_sbom_dir(sbom_dir)
CMake: Add API to allow referencing external packages in the SPDX SBOM This change modifies the existing implementation of the external references API to have better named options and better behavior. It also adds comments and exposes an internal API wrapper around the existing implementation. It's meant to be used by qtwebengine to reference the Chromium-generated SBOM. As a drive-by, it removes the previously unused RENAME option. A sample usage could be: qt_internal_sbom_get_external_document_ref_spdx_id( "qt5compat" document_ref_spdx_id) qt_internal_sbom_add_external_reference( EXTERNAL_DOCUMENT_FILE_PATH "/path/to/qt5compat-6.8.1.spdx.json" EXTERNAL_DOCUMENT_SPDX_ID "${document_ref_spdx_id}" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I13d0fe4d803449bec42f8b454c8131e4d727669a Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Moss Heim <moss.heim@qt.io>
2024-12-02 18:16:30 +01:00
set(ext_ref_sbom "${sbom_dir}/${arg_EXTERNAL_DOCUMENT_SPDX_ID}.cmake")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
file(GENERATE OUTPUT "${ext_ref_sbom}" CONTENT "${content}")
set_property(GLOBAL APPEND PROPERTY _qt_sbom_cmake_end_include_files "${ext_ref_sbom}")
endfunction()
# Helper to add info about a package to the sbom. Usually a package is a mapping to a cmake target.
function(_qt_internal_sbom_generate_add_package)
set(opt_args
CONTAINS_FILES
)
set(single_args
PACKAGE
VERSION
LICENSE_DECLARED
LICENSE_CONCLUDED
COPYRIGHT
DOWNLOAD_LOCATION
RELATIONSHIP
SPDXID
SUPPLIER
PURPOSE
COMMENT
)
set(multi_args
EXTREF
CPE
)
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value_and_error_if_empty(PACKAGE "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
set(check_option "")
if(arg_SPDXID)
set(check_option "CHECK" "${arg_SPDXID}")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_get_and_check_spdx_id(
VARIABLE arg_SPDXID
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
${check_option}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
HINTS "SPDXRef-${arg_PACKAGE}"
)
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value(DOWNLOAD_LOCATION "NOASSERTION")
_qt_internal_sbom_set_default_option_value(VERSION "unknown")
_qt_internal_sbom_set_default_option_value(SUPPLIER "Person: Anonymous")
_qt_internal_sbom_set_default_option_value(LICENSE_DECLARED "NOASSERTION")
_qt_internal_sbom_set_default_option_value(LICENSE_CONCLUDED "NOASSERTION")
_qt_internal_sbom_set_default_option_value(COPYRIGHT "NOASSERTION")
_qt_internal_sbom_set_default_option_value(PURPOSE "OTHER")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(fields "")
if(arg_LICENSE_CONCLUDED)
set(fields "${fields}
PackageLicenseConcluded: ${arg_LICENSE_CONCLUDED}"
)
else()
set(fields "${fields}
PackageLicenseConcluded: NOASSERTION"
)
endif()
if(arg_LICENSE_DECLARED)
set(fields "${fields}
PackageLicenseDeclared: ${arg_LICENSE_DECLARED}"
)
else()
set(fields "${fields}
PackageLicenseDeclared: NOASSERTION"
)
endif()
foreach(ext_ref IN LISTS arg_EXTREF)
set(fields "${fields}
ExternalRef: ${ext_ref}"
)
endforeach()
if(arg_CONTAINS_FILES)
set(fields "${fields}
FilesAnalyzed: true"
)
else()
set(fields "${fields}
FilesAnalyzed: false"
)
endif()
if(arg_COPYRIGHT)
set(fields "${fields}
PackageCopyrightText: ${arg_COPYRIGHT}"
)
else()
set(fields "${fields}
PackageCopyrightText: NOASSERTION"
)
endif()
if(arg_PURPOSE)
set(fields "${fields}
PrimaryPackagePurpose: ${arg_PURPOSE}"
)
else()
set(fields "${fields}
PrimaryPackagePurpose: OTHER"
)
endif()
if(arg_COMMENT)
set(fields "${fields}
PackageComment: ${arg_COMMENT}"
)
endif()
foreach(cpe IN LISTS arg_CPE)
set(fields "${fields}
ExternalRef: SECURITY cpe23Type ${cpe}"
)
endforeach()
CMake: Fix project spdx id used in implicit relationships We shouldn't be using the direct project name in spdx relationships, but rather the sanitized name which is prefixed with 'SPDXRef-Package-', to ensure sbom validation succeeds. Also we should default PROJECT_FOR_SPDX_ID to "Package-${arg_PROJECT}" if it's not set. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I354babcf4fea5f6efd9b32422dd8d3835ef50f15 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-08 15:37:41 +01:00
get_property(sbom_project_spdx_id GLOBAL PROPERTY _qt_sbom_project_spdx_id)
if(NOT sbom_project_spdx_id)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
message(FATAL_ERROR "Call _qt_internal_sbom_begin_project() first")
endif()
if(NOT arg_RELATIONSHIP)
CMake: Fix project spdx id used in implicit relationships We shouldn't be using the direct project name in spdx relationships, but rather the sanitized name which is prefixed with 'SPDXRef-Package-', to ensure sbom validation succeeds. Also we should default PROJECT_FOR_SPDX_ID to "Package-${arg_PROJECT}" if it's not set. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I354babcf4fea5f6efd9b32422dd8d3835ef50f15 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-08 15:37:41 +01:00
set(arg_RELATIONSHIP "${sbom_project_spdx_id} CONTAINS ${arg_SPDXID}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
else()
string(REPLACE "@QT_SBOM_LAST_SPDXID@" "${arg_SPDXID}" arg_RELATIONSHIP "${arg_RELATIONSHIP}")
endif()
_qt_internal_get_staging_area_spdx_file_path(staging_area_spdx_file)
set(content "
file(APPEND \"${staging_area_spdx_file}\"
\"
PackageName: ${arg_PACKAGE}
SPDXID: ${arg_SPDXID}
PackageDownloadLocation: ${arg_DOWNLOAD_LOCATION}
PackageVersion: ${arg_VERSION}
PackageSupplier: ${arg_SUPPLIER}${fields}
Relationship: ${arg_RELATIONSHIP}
\"
)
")
_qt_internal_get_current_project_sbom_dir(sbom_dir)
set(package_sbom "${sbom_dir}/${arg_SPDXID}.cmake")
file(GENERATE OUTPUT "${package_sbom}" CONTENT "${content}")
set_property(GLOBAL APPEND PROPERTY _qt_sbom_cmake_include_files "${package_sbom}")
endfunction()
CMake: Allow adding custom SBOM relationships to targets and projects Add a new SBOM_RELATIONSHIPS option to qt_internal_extend_target and friends that allows adding custom relationships to the current target. Add a new function qt_internal_sbom_add_project_relationship that allows adding custom relationships to the current project SBOM document. A sample usage might be: qt_internal_sbom_get_project_spdx_id(project_spdx_id) qt_internal_sbom_get_target_spdx_id(Svg svg_spdx_id) qt_internal_extend_target(Svg SBOM_RELATIONSHIPS "${svg_spdx_id} DESCENDANT_OF ${project_spdx_id}" ) qt_internal_sbom_add_project_relationship( RELATIONSHIPS "${svg_spdx_id} CONTAINS NOASSERTION" "${svg_spdx_id} DESCRIBES NOASSERTION" "${project_spdx_id} DESCRIBES NOASSERTION" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ie0119ca71b047c7515e1deaf84a5a67ea01b5274 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 19:13:29 +01:00
# Helper to add relationship entries to the current project SBOM document package.
#
# RELATIONSHIPS: A list of relationship strings to add to the current project relationships.
#
# Care must be taken to call the function right after project creation, before other targets are
# created, otherwise the relationship strings might be added to the wrong package.
# It doesn't seem to cause tooling to fail, but it's something to look out for.
function(_qt_internal_sbom_generate_add_project_relationship)
if(NOT QT_GENERATE_SBOM)
return()
endif()
set(opt_args "")
set(single_args "")
set(multi_args
RELATIONSHIPS
)
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value_and_error_if_empty(RELATIONSHIPS "")
CMake: Allow adding custom SBOM relationships to targets and projects Add a new SBOM_RELATIONSHIPS option to qt_internal_extend_target and friends that allows adding custom relationships to the current target. Add a new function qt_internal_sbom_add_project_relationship that allows adding custom relationships to the current project SBOM document. A sample usage might be: qt_internal_sbom_get_project_spdx_id(project_spdx_id) qt_internal_sbom_get_target_spdx_id(Svg svg_spdx_id) qt_internal_extend_target(Svg SBOM_RELATIONSHIPS "${svg_spdx_id} DESCENDANT_OF ${project_spdx_id}" ) qt_internal_sbom_add_project_relationship( RELATIONSHIPS "${svg_spdx_id} CONTAINS NOASSERTION" "${svg_spdx_id} DESCRIBES NOASSERTION" "${project_spdx_id} DESCRIBES NOASSERTION" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ie0119ca71b047c7515e1deaf84a5a67ea01b5274 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 19:13:29 +01:00
_qt_internal_get_staging_area_spdx_file_path(staging_area_spdx_file)
get_property(counter GLOBAL PROPERTY _qt_sbom_relationship_counter)
set(current_counter "${counter}")
math(EXPR counter "${counter} + 1")
set_property(GLOBAL PROPERTY _qt_sbom_relationship_counter "${counter}")
set(relationships "${arg_RELATIONSHIPS}")
list(REMOVE_DUPLICATES relationships)
list(JOIN relationships "\nRelationship: " relationships)
set(content "
# Custom relationship index: ${current_counter}
file(APPEND \"${staging_area_spdx_file}\"
\"
Relationship: ${relationships}\")
")
_qt_internal_get_current_project_sbom_dir(sbom_dir)
set(ext_ref_sbom "${sbom_dir}/relationship_${counter}.cmake")
file(GENERATE OUTPUT "${ext_ref_sbom}" CONTENT "${content}")
set_property(GLOBAL APPEND PROPERTY _qt_sbom_cmake_include_files "${ext_ref_sbom}")
endfunction()
CMake: Add API to include custom cmake files during SBOM generation Allow including custom cmake files at predefined steps of the sbom generation using the new internal qt_internal_sbom_add_cmake_include_step API. This covers pre-existing steps like BEGIN, END, POST_GENERATION, and VERIFY, as well as two new steps, BEFORE_CHECKSUM and AFTER_CHECKSUM. A sample usage of the new API could be: set(step_path "${CMAKE_CURRENT_BINARY_DIR}/run_some_cmake.cmake") file(WRITE "${step_path}" " # This is a CMake script to run some CMake code. message(STATUS \"Running some CMake code...\") message(STATUS \"Done running some CMake code.\") " ) foreach(step IN ITEMS BEGIN END BEFORE_CHECKSUM AFTER_CHECKSUM POST_GENERATION VERIFY) qt_internal_sbom_add_cmake_include_step( STEP "${step}" INCLUDE_PATH "${step_path}" ) endforeach() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: I6f9512bf3a3bc3ebeb7d21426e5a3833280e42a5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 17:38:31 +01:00
# Adds a cmake include file to the sbom generation process at a specific step.
# INCLUDE_PATH - path to the cmake file to include.
# STEP - one of
# BEGIN
# END
# POST_GENERATION
# VERIFY
#
# BEGIN includes are included after the spdx staging file is created.
#
# END includes are included after the all the BEGIN ones are included.
#
# BEFORE_CHECKSUM includes are included after the verification codes have been collected
# and before their merged checksum(s) has been computed. Only relevant for install time sboms.
#
# AFTER_CHECKSUM includes are included after the checksum has been computed and written to the
# QT_SBOM_VERIFICATION_CODE variable. Only relevant for install time sboms.
#
# POST_GENERATION includes are included for both build and install time sboms, after
# sbom generation has finished.
# Currently used for adding reuse lint and reuse source steps, before VERIFY include are run.
#
# VERIFY includes only make sense on installation, where the checksums are present, so they are
# only included during install time.
# Used for generating a json sbom from the tag value one, for running ntia compliance check, etc.
function(_qt_internal_sbom_add_cmake_include_step)
set(opt_args "")
set(single_args
STEP
INCLUDE_PATH
)
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
if(NOT arg_STEP)
message(FATAL_ERROR "STEP is required")
endif()
if(NOT arg_INCLUDE_PATH)
message(FATAL_ERROR "INCLUDE_PATH is required")
endif()
set(step "${arg_STEP}")
if(step STREQUAL "BEGIN")
set(property "_qt_sbom_cmake_include_files")
elseif(step STREQUAL "END")
set(property "_qt_sbom_cmake_end_include_files")
elseif(step STREQUAL "BEFORE_CHECKSUM")
set(property "_qt_sbom_cmake_before_checksum_include_files")
elseif(step STREQUAL "AFTER_CHECKSUM")
set(property "_qt_sbom_cmake_after_checksum_include_files")
elseif(step STREQUAL "POST_GENERATION")
set(property "_qt_sbom_cmake_post_generation_include_files")
elseif(step STREQUAL "VERIFY")
set(property "_qt_sbom_cmake_verify_include_files")
else()
message(FATAL_ERROR "Invalid SBOM cmake include step name: ${step}")
endif()
set_property(GLOBAL APPEND PROPERTY "${property}" "${arg_INCLUDE_PATH}")
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Helper to add a license text from a file or text into the sbom document.
function(_qt_internal_sbom_generate_add_license)
set(opt_args "")
set(single_args
LICENSE_ID
EXTRACTED_TEXT
)
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value_and_error_if_empty(LICENSE_ID "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
set(check_option "")
if(arg_SPDXID)
set(check_option "CHECK" "${arg_SPDXID}")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_get_and_check_spdx_id(
VARIABLE arg_SPDXID
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
${check_option}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
HINTS "SPDXRef-${arg_LICENSE_ID}"
)
if(NOT arg_EXTRACTED_TEXT)
set(licenses_dir "${PROJECT_SOURCE_DIR}/LICENSES")
file(READ "${licenses_dir}/${arg_LICENSE_ID}.txt" arg_EXTRACTED_TEXT)
string(PREPEND arg_EXTRACTED_TEXT "<text>")
string(APPEND arg_EXTRACTED_TEXT "</text>")
endif()
_qt_internal_get_staging_area_spdx_file_path(staging_area_spdx_file)
set(content "
file(APPEND \"${staging_area_spdx_file}\"
\"
LicenseID: ${arg_LICENSE_ID}
ExtractedText: ${arg_EXTRACTED_TEXT}
\"
)
")
_qt_internal_get_current_project_sbom_dir(sbom_dir)
set(license_sbom "${sbom_dir}/${arg_SPDXID}.cmake")
file(GENERATE OUTPUT "${license_sbom}" CONTENT "${content}")
set_property(GLOBAL APPEND PROPERTY _qt_sbom_cmake_end_include_files "${license_sbom}")
endfunction()
# Helper to retrieve a valid spdx id, given some hints.
# HINTS can be a list of values, one of which will be sanitized and used as the spdx id.
# CHECK is expected to be a valid spdx id.
function(_qt_internal_sbom_get_and_check_spdx_id)
set(opt_args "")
set(single_args
VARIABLE
CHECK
)
set(multi_args
HINTS
)
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Prepend underscore to qt_internal_sbom_set_default_option_value Because it's in a Public cmake file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: If3f92af848193cb824b88e93e17965ffed93cf2b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 16:41:33 +01:00
_qt_internal_sbom_set_default_option_value_and_error_if_empty(VARIABLE "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(NOT arg_CHECK)
get_property(spdx_id_count GLOBAL PROPERTY _qt_sbom_spdx_id_count)
set(suffix "-${spdx_id_count}")
math(EXPR spdx_id_count "${spdx_id_count} + 1")
set_property(GLOBAL PROPERTY _qt_sbom_spdx_id_count "${spdx_id_count}")
foreach(hint IN LISTS arg_HINTS)
_qt_internal_sbom_get_sanitized_spdx_id(id "${hint}")
if(id)
set(id "${id}${suffix}")
break()
endif()
endforeach()
if(NOT id)
set(id "SPDXRef${suffix}")
endif()
else()
set(id "${arg_CHECK}")
endif()
if("${id}" MATCHES "^SPDXRef-[-]+$"
OR (NOT "${id}" MATCHES "^SPDXRef-[-a-zA-Z0-9]+$"))
message(FATAL_ERROR "Invalid SPDXID \"${id}\"")
endif()
set(${arg_VARIABLE} "${id}" PARENT_SCOPE)
endfunction()
CMake: Rework SBOM python dependency lookup and handling Introduce a bunch of helper functions to split out the finding of a specific python interpreter, a specific python dependency within a given python interpreter, and the handling of dependencies for each of the possible SBOM operations: GENERATE_JSON, VERIFY_SBOM, and RUN_NTIA. In a future change this will allow us to conditionally enable certain operations, depending on which dependencies are available, without failing the build outright. One behavior change is better logging of what goes wrong if a dependency is missing. Another is the double look up of python on macOS, to account for the system framework python not having the required dependencies. These was previously an internal opt in, but now it is done automatically. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3054f2649c0092dc5f2d3e299065f0f62dc6c5fb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-16 15:21:34 +02:00
Reference in New Issue Copy Permalink