1berry/qtbase
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity
qtbase/cmake/QtPublicSbomHelpers.cmake

1868 lines
66 KiB
CMake
Raw Permalink Normal View History

CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Copyright (C) 2024 The Qt Company Ltd.
# SPDX-License-Identifier: BSD-3-Clause
# Starts repo sbom generation.
# Should be called before any targets are added to the sbom.
#
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
# INSTALL_PREFIX should be passed a value like CMAKE_INSTALL_PREFIX or QT_STAGING_PREFIX.
# The default value is \${CMAKE_INSTALL_PREFIX}, which is evaluated at install time, not configure
# time.
# This default value is the /preferred/ value, to ensure using cmake --install . --prefix <path>
# works correctly for lookup of installed files during SBOM generation.
#
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# INSTALL_SBOM_DIR should be passed a value like CMAKE_INSTALL_DATAROOTDIR or
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
# Qt's INSTALL_SBOMDIR.
# The default value is "sbom".
#
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# SUPPLIER, SUPPLIER_URL, DOCUMENT_NAMESPACE, COPYRIGHTS are self-explanatory.
function(_qt_internal_sbom_begin_project)
CMake: Allow force disabling generation of SBOM We need it to disable generation for certain repos like qtqa, even if the platform config says that SBOMs should be enabled. We can't just set QT_GENERATE_SBOM to OFF, because the general SBOM CI instructions are added later that module specific ones, and thus would override the value back to ON. Providing a separate internal variable allows us to disable it with a higher priority. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: If7803ae4aac0886d605a542e3f05ad9533bb8108 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-06-21 15:23:33 +02:00
# Allow opt out via an internal variable. Will be used in CI for repos like qtqa.
if(QT_INTERNAL_FORCE_NO_GENERATE_SBOM)
set(QT_GENERATE_SBOM OFF CACHE BOOL "Generate SBOM" FORCE)
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(NOT QT_GENERATE_SBOM)
return()
endif()
CMake: Add a fake deterministic SBOM generation option Replaces content like timestamps, version strings and checksums with constant values to make the SBOM generation deterministic. This is useful for reproducible builds, to allow cleaner inter-diffs while developing new SBOM features. Can be enabled by configuring with -DQT_SBOM_FAKE_DETERMINISTIC_BUILD=ON Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Id0003bdd23fe57abd70213a2a108885861693242 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 10:05:05 +01:00
_qt_internal_sbom_setup_fake_deterministic_build()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(opt_args
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
USE_GIT_VERSION
__QT_INTERNAL_HANDLE_QT_REPO
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
set(single_args
INSTALL_PREFIX
INSTALL_SBOM_DIR
LICENSE_EXPRESSION
SUPPLIER
SUPPLIER_URL
DOWNLOAD_LOCATION
DOCUMENT_NAMESPACE
VERSION
SBOM_PROJECT_NAME
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
QT_REPO_PROJECT_NAME
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CPE
)
set(multi_args
COPYRIGHTS
)
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
if(CMAKE_VERSION LESS_EQUAL "3.19")
if(QT_IGNORE_MIN_CMAKE_VERSION_FOR_SBOM)
message(STATUS
"Using CMake version older than 3.19, and QT_IGNORE_MIN_CMAKE_VERSION_FOR_SBOM was "
"set to ON. qt_attribution.json files will not be processed.")
else()
message(FATAL_ERROR
"Generating an SBOM requires CMake version 3.19 or newer. You can pass "
"-DQT_IGNORE_MIN_CMAKE_VERSION_FOR_SBOM=ON to try to generate the SBOM anyway, "
"but it is not officially supported, and the SBOM might be incomplete.")
endif()
endif()
# The ntia-conformance-checker insists that a SPDX document contain at least one
# relationship that DESCRIBES a package, and that the package contains the string
# "Package-" in the spdx id. boot2qt spdx seems to contain the same.
if(arg_SBOM_PROJECT_NAME)
_qt_internal_sbom_set_root_project_name("${arg_SBOM_PROJECT_NAME}")
else()
_qt_internal_sbom_set_root_project_name("${PROJECT_NAME}")
endif()
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
if(arg_QT_REPO_PROJECT_NAME)
_qt_internal_sbom_set_qt_repo_project_name("${arg_QT_REPO_PROJECT_NAME}")
else()
_qt_internal_sbom_set_qt_repo_project_name("${PROJECT_NAME}")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_get_root_project_name_for_spdx_id(repo_project_name_for_spdx_id)
_qt_internal_sbom_get_root_project_name_lower_case(repo_project_name_lowercase)
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
set(repo_supplier_url "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(arg_SUPPLIER_URL)
set(repo_supplier_url "${arg_SUPPLIER_URL}")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
elseif(arg___QT_INTERNAL_HANDLE_QT_REPO)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_get_default_supplier_url(repo_supplier_url)
endif()
# Manual override.
if(arg_VERSION)
set(QT_SBOM_GIT_VERSION "${arg_VERSION}")
set(QT_SBOM_GIT_VERSION_PATH "${arg_VERSION}")
set(QT_SBOM_GIT_HASH "") # empty on purpose, no source of info
CMake: Expose the short git hash during SBOM generation Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id87afba7bce4d67984c7e8811444fd35be758bec Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-07-24 17:29:11 +02:00
set(QT_SBOM_GIT_HASH_SHORT "") # empty on purpose, no source of info
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(non_git_version "${arg_VERSION}")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
elseif(arg_USE_GIT_VERSION)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Query git version info.
_qt_internal_find_git_package()
_qt_internal_query_git_version(
EMPTY_VALUE_WHEN_NOT_GIT_REPO
OUT_VAR_PREFIX __sbom_
)
set(QT_SBOM_GIT_VERSION "${__sbom_git_version}")
set(QT_SBOM_GIT_VERSION_PATH "${__sbom_git_version_path}")
set(QT_SBOM_GIT_HASH "${__sbom_git_hash}")
CMake: Expose the short git hash during SBOM generation Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id87afba7bce4d67984c7e8811444fd35be758bec Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-07-24 17:29:11 +02:00
set(QT_SBOM_GIT_HASH_SHORT "${__sbom_git_hash_short}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Git version might not be available.
set(non_git_version "${QT_REPO_MODULE_VERSION}")
if(NOT QT_SBOM_GIT_VERSION)
set(QT_SBOM_GIT_VERSION "${non_git_version}")
endif()
if(NOT QT_SBOM_GIT_VERSION_PATH)
set(QT_SBOM_GIT_VERSION_PATH "${non_git_version}")
endif()
endif()
CMake: Fix sbom git vars not being available in various scopes Initially the git vars were assigned to the parent scope of the _qt_internal_sbom_begin_project function, with the intent to set them in the global scope. But the function was later wrapped in other functions, so the variables stopped being accessible. Instead of playing with recursive PARENT_SCOPEs, save the variables in global properties like we do for other info, and use a new _qt_internal_sbom_get_git_version_vars() function to query the vars in the code that needs them. This fixes generated purls to contain the git version and hashes. Also add a new internal API wrapper macro called qt_internal_sbom_get_git_version_vars to allow calling it in other repos. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I061b34f418c1ecc1c66c8c01ef758d2f40611ede Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 11:46:36 +01:00
# Save the variables in a global property to later query them in other functions.
set_property(GLOBAL PROPERTY QT_SBOM_GIT_VERSION "${QT_SBOM_GIT_VERSION}")
set_property(GLOBAL PROPERTY QT_SBOM_GIT_VERSION_PATH "${QT_SBOM_GIT_VERSION_PATH}")
set_property(GLOBAL PROPERTY QT_SBOM_GIT_HASH "${QT_SBOM_GIT_HASH}")
set_property(GLOBAL PROPERTY QT_SBOM_GIT_HASH_SHORT "${QT_SBOM_GIT_HASH_SHORT}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(arg_DOCUMENT_NAMESPACE)
set(repo_spdx_namespace "${arg_DOCUMENT_NAMESPACE}")
else()
CMake: Add a few more internal sbom helper functions Add a few internal functions to allow - getting the sbom project supplier - project supplier url - project namespace - computing a project namespace - computing a project file name - getting a sanitized spdx id given a hint Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I0dc3df274eaf6882a6af021aabee75501b5083f8 Reviewed-by: Moss Heim <moss.heim@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-21 19:32:35 +01:00
_qt_internal_sbom_compute_project_namespace(repo_spdx_namespace
PROJECT_NAME "${repo_project_name_lowercase}"
SUPPLIER_URL "${repo_supplier_url}"
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
if(arg_INSTALL_SBOM_DIR)
set(install_sbom_dir "${arg_INSTALL_SBOM_DIR}")
elseif(INSTALL_SBOMDIR)
set(install_sbom_dir "${INSTALL_SBOMDIR}")
else()
set(install_sbom_dir "sbom")
endif()
if(arg_INSTALL_PREFIX)
set(install_prefix "${arg_INSTALL_PREFIX}")
else()
# The variable is escaped, so it is evaluated during cmake install time, so that the value
# can be overridden with cmake --install . --prefix <path>.
set(install_prefix "\${CMAKE_INSTALL_PREFIX}")
endif()
CMake: Add a few more internal sbom helper functions Add a few internal functions to allow - getting the sbom project supplier - project supplier url - project namespace - computing a project namespace - computing a project file name - getting a sanitized spdx id given a hint Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I0dc3df274eaf6882a6af021aabee75501b5083f8 Reviewed-by: Moss Heim <moss.heim@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-21 19:32:35 +01:00
_qt_internal_sbom_compute_project_file_name(repo_project_file_name
PROJECT_NAME "${repo_project_name_lowercase}"
VERSION_SUFFIX "${non_git_version}"
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(repo_spdx_relative_install_path
CMake: Add a few more internal sbom helper functions Add a few internal functions to allow - getting the sbom project supplier - project supplier url - project namespace - computing a project namespace - computing a project file name - getting a sanitized spdx id given a hint Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I0dc3df274eaf6882a6af021aabee75501b5083f8 Reviewed-by: Moss Heim <moss.heim@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-21 19:32:35 +01:00
"${arg_INSTALL_SBOM_DIR}/${repo_project_file_name}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Prepend DESTDIR, to allow relocating installed sbom. Needed for CI.
set(repo_spdx_install_path
CMake: Support cmake --install . --prefix <path> for SBOM generation Before this change, the SBOM generation only considered the CMAKE_INSTALL_PREFIX set at configure time for the purposes of file checksum computation, external document lookup, and SBOM file installation. When cmake --install . --prefix <path> is used, the install time CMAKE_INSTALL_PREFIX variable is overridden with the provided path, and can be different from the configure time value. This path was applied to the installation of regular files, libraries, etc, but not to the SBOM generation. This caused issues like not being able to find a library to compute its checksum. Instead of hard-coding the value to QT_STAGING_PREFIX, just don't pass a value at all, and rely on the new default of using an install-time-evaluated \${CMAKE_INSTALL_PREFIX}. Keep the ability of specifying a custom prefix just in case. Modify all the code that used a hard-coded CMAKE_INSTALL_PREFIX to use a install-time-evaluated one instead. As a drive-by, also set a proper default value for the INSTALL_SBOM_DIR option. Pick-to: 6.8 Fixes: QTBUG-131883 Change-Id: Ifde6ab282ac40f10c5bf51976121065c7dc631eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-04 17:49:59 +01:00
"\$ENV{DESTDIR}${install_prefix}/${repo_spdx_relative_install_path}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(arg_LICENSE_EXPRESSION)
set(repo_license "${arg_LICENSE_EXPRESSION}")
else()
# Default to NOASSERTION for root repo SPDX packages, because we have some repos
# with multiple licenses and AND-ing them together will create a giant unreadable list.
# It's better to rely on the more granular package licenses.
set(repo_license "")
endif()
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
set(repo_license_option "")
if(repo_license)
set(repo_license_option "LICENSE" "${repo_license}")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(arg_COPYRIGHTS)
list(JOIN arg_COPYRIGHTS "\n" arg_COPYRIGHTS)
set(repo_copyright "<text>${arg_COPYRIGHTS}</text>")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
elseif(arg___QT_INTERNAL_HANDLE_QT_REPO)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_get_default_qt_copyright_header(repo_copyright)
endif()
if(arg_SUPPLIER)
set(repo_supplier "${arg_SUPPLIER}")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
elseif(arg___QT_INTERNAL_HANDLE_QT_REPO)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_get_default_supplier(repo_supplier)
endif()
if(arg_CPE)
set(qt_cpe "${arg_CPE}")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
elseif(arg___QT_INTERNAL_HANDLE_QT_REPO)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_get_cpe_qt_repo(qt_cpe)
else()
set(qt_cpe "")
endif()
if(arg_DOWNLOAD_LOCATION)
set(download_location "${arg_DOWNLOAD_LOCATION}")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
elseif(arg___QT_INTERNAL_HANDLE_QT_REPO)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_get_qt_repo_source_download_location(download_location)
endif()
CMake: When available add the repo configure line to the SBOM When Qt is configured using the configure script, rather than directly via CMake, we can obtain the full list of configure arguments passed from the config.opt file. Add the configure line to the repo project SBOM package comment field when it is available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0c5554dcda2f71ec4ba034b14c82a99757dc790c Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-17 17:20:29 +02:00
set(project_comment "")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg___QT_INTERNAL_HANDLE_QT_REPO)
_qt_internal_get_configure_line(configure_line)
if(configure_line)
set(configure_line_comment
"\n${repo_project_name_lowercase} was configured with:\n ${configure_line}\n")
string(APPEND project_comment "${configure_line_comment}")
endif()
CMake: When available add the repo configure line to the SBOM When Qt is configured using the configure script, rather than directly via CMake, we can obtain the full list of configure arguments passed from the config.opt file. Add the configure line to the repo project SBOM package comment field when it is available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0c5554dcda2f71ec4ba034b14c82a99757dc790c Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-17 17:20:29 +02:00
endif()
CMake: Fix SBOM PROJECT_COMMENT CMP0174 related warning We should only pass PROJECT_COMMENT if it has a value. We also need to escape semicolons, because the project comment might contain the qt configure line, and that might have passed arguments like -qpa offscreen\;xcb Pick-to: 6.8 6.9 Change-Id: I934cf75c376b3466ba91a433d009e6eaa77a60fa Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-03-05 11:05:10 +01:00
if(project_comment)
# Escape any potential semicolons.
string(REPLACE ";" "\\;" project_comment "${project_comment}")
set(project_comment PROJECT_COMMENT "${project_comment}")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_begin_project_generate(
OUTPUT "${repo_spdx_install_path}"
CMake: Make ninja 'sbom' work for top-level builds We need to pass all dirs where sboms will be generated, for external document referencing. We also need to set up dependencies between the repo sbom custom targets, so that qtsvg sbom is only run after qtbase sbom. We use the dependencies.yaml info to set up the dependencies. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id3331e11742bc2c86e7ed52ce26b3ff21eace359 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-20 19:21:53 +02:00
OUTPUT_RELATIVE_PATH "${repo_spdx_relative_install_path}"
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
${repo_license_option}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
COPYRIGHT "${repo_copyright}"
SUPPLIER "${repo_supplier}" # This must not contain spaces!
SUPPLIER_URL "${repo_supplier_url}"
DOWNLOAD_LOCATION "${download_location}"
PROJECT "${repo_project_name_lowercase}"
CMake: Fix SBOM PROJECT_COMMENT CMP0174 related warning We should only pass PROJECT_COMMENT if it has a value. We also need to escape semicolons, because the project comment might contain the qt configure line, and that might have passed arguments like -qpa offscreen\;xcb Pick-to: 6.8 6.9 Change-Id: I934cf75c376b3466ba91a433d009e6eaa77a60fa Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-03-05 11:05:10 +01:00
${project_comment}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
PROJECT_FOR_SPDX_ID "${repo_project_name_for_spdx_id}"
NAMESPACE "${repo_spdx_namespace}"
CPE "${qt_cpe}"
OUT_VAR_PROJECT_SPDX_ID repo_project_spdx_id
)
CMake: Add configure-time dependency on project attribution files Otherwise the SBOM is not regenerated if the attribution files are modified. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I5b3f62e254aa70021ed06fac73f881bcbb110c31 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-09-03 12:56:40 +02:00
set_property(GLOBAL PROPERTY _qt_internal_project_attribution_files "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set_property(GLOBAL PROPERTY _qt_internal_sbom_repo_document_namespace
"${repo_spdx_namespace}")
set_property(GLOBAL PROPERTY _qt_internal_sbom_relative_installed_repo_document_path
"${repo_spdx_relative_install_path}")
set_property(GLOBAL PROPERTY _qt_internal_sbom_repo_project_name_lowercase
"${repo_project_name_lowercase}")
set_property(GLOBAL PROPERTY _qt_internal_sbom_install_prefix
"${arg_INSTALL_PREFIX}")
set_property(GLOBAL PROPERTY _qt_internal_sbom_project_spdx_id
"${repo_project_spdx_id}")
CMake: Fix standalone build of sqldrivers with SBOM The standalone build of the sqldrivers plugin never called the SBOM project begin and end functions. This cause an error in qt_internal_add_plugin which tried to read the SBOM project name. Replace the calls to qt_prepare_standalone_project and qt_print_feature_summary with qt_build_repo_begin and qt_build_repo_end. This ensures the SBOM project is setup, as well as many other behaviors that a standalone internal build of a module is expected to have. Additionally we need to tell the SBOM project where to find the licenses for the standalone build. This is done by setting the new QT_SBOM_LICENSE_DIRS variable to the qtbase license directory. Pick-to: 6.8 Fixes: QTBUG-131799 Change-Id: I2e31ecffdff28561d1c4a6b8fbcd8125188d2c48 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-03 11:56:54 +01:00
# Collect project licenses.
set(license_dirs "")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg___QT_INTERNAL_HANDLE_QT_REPO AND EXISTS "${PROJECT_SOURCE_DIR}/LICENSES")
CMake: Fix standalone build of sqldrivers with SBOM The standalone build of the sqldrivers plugin never called the SBOM project begin and end functions. This cause an error in qt_internal_add_plugin which tried to read the SBOM project name. Replace the calls to qt_prepare_standalone_project and qt_print_feature_summary with qt_build_repo_begin and qt_build_repo_end. This ensures the SBOM project is setup, as well as many other behaviors that a standalone internal build of a module is expected to have. Additionally we need to tell the SBOM project where to find the licenses for the standalone build. This is done by setting the new QT_SBOM_LICENSE_DIRS variable to the qtbase license directory. Pick-to: 6.8 Fixes: QTBUG-131799 Change-Id: I2e31ecffdff28561d1c4a6b8fbcd8125188d2c48 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-03 11:56:54 +01:00
list(APPEND license_dirs "${PROJECT_SOURCE_DIR}/LICENSES")
endif()
# Allow specifying extra license dirs via a variable. Useful for standalone projects
# like sqldrivers.
if(QT_SBOM_LICENSE_DIRS)
foreach(license_dir IN LISTS QT_SBOM_LICENSE_DIRS)
if(EXISTS "${license_dir}")
list(APPEND license_dirs "${license_dir}")
endif()
endforeach()
endif()
list(REMOVE_DUPLICATES license_dirs)
set(license_file_wildcard "LicenseRef-*.txt")
list(TRANSFORM license_dirs APPEND "/${license_file_wildcard}" OUTPUT_VARIABLE license_globs)
file(GLOB license_files ${license_globs})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
foreach(license_file IN LISTS license_files)
get_filename_component(license_id "${license_file}" NAME_WLE)
_qt_internal_sbom_add_license(
LICENSE_ID "${license_id}"
LICENSE_PATH "${license_file}"
NO_LICENSE_REF_PREFIX
)
endforeach()
# Make sure that any system library dependencies that have been found via qt_find_package or
# _qt_internal_find_third_party_dependencies have their spdx id registered now.
_qt_internal_sbom_record_system_library_spdx_ids()
set_property(GLOBAL PROPERTY _qt_internal_sbom_repo_begin_called TRUE)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
_qt_internal_sbom_setup_project_ops()
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
# Check various internal options to decide which sbom generation operations should be setup.
# Considered operations are generation of a JSON sbom, validation of the SBOM, NTIA checker, etc.
function(_qt_internal_sbom_setup_project_ops)
set(options "")
CMake: Change SBOM generation to be enabled by default (mostly) Previously SBOM generation was opt-in. This patch changes the generation of the plain-text tag:value SBOM to be enabled by default, except for: - developer builds - no-prefix builds - standalone tests or examples - cmake build tests The JSON SBOM generation and the verification steps have also been changed to be enabled by default, but only if the Python dependencies can be found. If the dependencies are not found, the build will skip the generation and verification steps. Four new configure options have been added to control these aspects: -(no-)sbom-json: Allows explicitly enabling or disabling JSON SBOM generation -(no-)sbom-json-required: Fails the build if JSON SBOM generation Python dependencies are not found -(no-)sbom-verify: Allows explicitly enabling or disabling SBOM verification -(no-)sbom-verify-required: Fails the build if SBOM verification Python dependencies are not found There are corresponding CMake variables for each of the configure options, see the cmake mapping document. [ChangeLog][Build Systems] SBOM generation is now enabled by default, when building Qt, except for developer builds and no-prefix builds. JSON SBOM generation is enabled by default if the required Python dependencies are available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6dbe1869f8342154a89ff2ab84ac53c9ef1b2eb7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-14 17:05:23 +02:00
if(QT_SBOM_GENERATE_JSON OR QT_INTERNAL_SBOM_GENERATE_JSON OR QT_INTERNAL_SBOM_DEFAULT_CHECKS)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options GENERATE_JSON)
CMake: Change SBOM generation to be enabled by default (mostly) Previously SBOM generation was opt-in. This patch changes the generation of the plain-text tag:value SBOM to be enabled by default, except for: - developer builds - no-prefix builds - standalone tests or examples - cmake build tests The JSON SBOM generation and the verification steps have also been changed to be enabled by default, but only if the Python dependencies can be found. If the dependencies are not found, the build will skip the generation and verification steps. Four new configure options have been added to control these aspects: -(no-)sbom-json: Allows explicitly enabling or disabling JSON SBOM generation -(no-)sbom-json-required: Fails the build if JSON SBOM generation Python dependencies are not found -(no-)sbom-verify: Allows explicitly enabling or disabling SBOM verification -(no-)sbom-verify-required: Fails the build if SBOM verification Python dependencies are not found There are corresponding CMake variables for each of the configure options, see the cmake mapping document. [ChangeLog][Build Systems] SBOM generation is now enabled by default, when building Qt, except for developer builds and no-prefix builds. JSON SBOM generation is enabled by default if the required Python dependencies are available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6dbe1869f8342154a89ff2ab84ac53c9ef1b2eb7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-14 17:05:23 +02:00
endif()
# Tring to generate the JSON might fail if the python dependencies are not available.
# The user can explicitly request to fail the build if dependencies are not found.
# error out. For internal options that the CI uses, we always want to fail the build if the
# deps are not found.
if(QT_SBOM_REQUIRE_GENERATE_JSON OR QT_INTERNAL_SBOM_GENERATE_JSON
OR QT_INTERNAL_SBOM_DEFAULT_CHECKS)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options GENERATE_JSON_REQUIRED)
CMake: Change SBOM generation to be enabled by default (mostly) Previously SBOM generation was opt-in. This patch changes the generation of the plain-text tag:value SBOM to be enabled by default, except for: - developer builds - no-prefix builds - standalone tests or examples - cmake build tests The JSON SBOM generation and the verification steps have also been changed to be enabled by default, but only if the Python dependencies can be found. If the dependencies are not found, the build will skip the generation and verification steps. Four new configure options have been added to control these aspects: -(no-)sbom-json: Allows explicitly enabling or disabling JSON SBOM generation -(no-)sbom-json-required: Fails the build if JSON SBOM generation Python dependencies are not found -(no-)sbom-verify: Allows explicitly enabling or disabling SBOM verification -(no-)sbom-verify-required: Fails the build if SBOM verification Python dependencies are not found There are corresponding CMake variables for each of the configure options, see the cmake mapping document. [ChangeLog][Build Systems] SBOM generation is now enabled by default, when building Qt, except for developer builds and no-prefix builds. JSON SBOM generation is enabled by default if the required Python dependencies are available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6dbe1869f8342154a89ff2ab84ac53c9ef1b2eb7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-14 17:05:23 +02:00
endif()
if(QT_SBOM_VERIFY OR QT_INTERNAL_SBOM_VERIFY OR QT_INTERNAL_SBOM_DEFAULT_CHECKS)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options VERIFY_SBOM)
CMake: Change SBOM generation to be enabled by default (mostly) Previously SBOM generation was opt-in. This patch changes the generation of the plain-text tag:value SBOM to be enabled by default, except for: - developer builds - no-prefix builds - standalone tests or examples - cmake build tests The JSON SBOM generation and the verification steps have also been changed to be enabled by default, but only if the Python dependencies can be found. If the dependencies are not found, the build will skip the generation and verification steps. Four new configure options have been added to control these aspects: -(no-)sbom-json: Allows explicitly enabling or disabling JSON SBOM generation -(no-)sbom-json-required: Fails the build if JSON SBOM generation Python dependencies are not found -(no-)sbom-verify: Allows explicitly enabling or disabling SBOM verification -(no-)sbom-verify-required: Fails the build if SBOM verification Python dependencies are not found There are corresponding CMake variables for each of the configure options, see the cmake mapping document. [ChangeLog][Build Systems] SBOM generation is now enabled by default, when building Qt, except for developer builds and no-prefix builds. JSON SBOM generation is enabled by default if the required Python dependencies are available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6dbe1869f8342154a89ff2ab84ac53c9ef1b2eb7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-14 17:05:23 +02:00
endif()
# Do the same requirement check for SBOM verification.
if(QT_SBOM_REQUIRE_VERIFY OR QT_INTERNAL_SBOM_VERIFY OR QT_INTERNAL_SBOM_DEFAULT_CHECKS)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options VERIFY_SBOM_REQUIRED)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Change SBOM generation to be enabled by default (mostly) Previously SBOM generation was opt-in. This patch changes the generation of the plain-text tag:value SBOM to be enabled by default, except for: - developer builds - no-prefix builds - standalone tests or examples - cmake build tests The JSON SBOM generation and the verification steps have also been changed to be enabled by default, but only if the Python dependencies can be found. If the dependencies are not found, the build will skip the generation and verification steps. Four new configure options have been added to control these aspects: -(no-)sbom-json: Allows explicitly enabling or disabling JSON SBOM generation -(no-)sbom-json-required: Fails the build if JSON SBOM generation Python dependencies are not found -(no-)sbom-verify: Allows explicitly enabling or disabling SBOM verification -(no-)sbom-verify-required: Fails the build if SBOM verification Python dependencies are not found There are corresponding CMake variables for each of the configure options, see the cmake mapping document. [ChangeLog][Build Systems] SBOM generation is now enabled by default, when building Qt, except for developer builds and no-prefix builds. JSON SBOM generation is enabled by default if the required Python dependencies are available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6dbe1869f8342154a89ff2ab84ac53c9ef1b2eb7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-14 17:05:23 +02:00
CMake: Split SBOM verification and NTIA compliance into separate ops This will allow us to run only the first, but not the second, if the second won't have it's dependencies met. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I141b4bd3b76a71495c760a118bdf1397ee7e16b5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-16 19:30:44 +02:00
if(QT_INTERNAL_SBOM_VERIFY_NTIA_COMPLIANT OR QT_INTERNAL_SBOM_DEFAULT_CHECKS)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options VERIFY_NTIA_COMPLIANT)
CMake: Split SBOM verification and NTIA compliance into separate ops This will allow us to run only the first, but not the second, if the second won't have it's dependencies met. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I141b4bd3b76a71495c760a118bdf1397ee7e16b5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-16 19:30:44 +02:00
endif()
CMake: Change SBOM generation to be enabled by default (mostly) Previously SBOM generation was opt-in. This patch changes the generation of the plain-text tag:value SBOM to be enabled by default, except for: - developer builds - no-prefix builds - standalone tests or examples - cmake build tests The JSON SBOM generation and the verification steps have also been changed to be enabled by default, but only if the Python dependencies can be found. If the dependencies are not found, the build will skip the generation and verification steps. Four new configure options have been added to control these aspects: -(no-)sbom-json: Allows explicitly enabling or disabling JSON SBOM generation -(no-)sbom-json-required: Fails the build if JSON SBOM generation Python dependencies are not found -(no-)sbom-verify: Allows explicitly enabling or disabling SBOM verification -(no-)sbom-verify-required: Fails the build if SBOM verification Python dependencies are not found There are corresponding CMake variables for each of the configure options, see the cmake mapping document. [ChangeLog][Build Systems] SBOM generation is now enabled by default, when building Qt, except for developer builds and no-prefix builds. JSON SBOM generation is enabled by default if the required Python dependencies are available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6dbe1869f8342154a89ff2ab84ac53c9ef1b2eb7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-14 17:05:23 +02:00
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(QT_INTERNAL_SBOM_SHOW_TABLE OR QT_INTERNAL_SBOM_DEFAULT_CHECKS)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options SHOW_TABLE)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Change SBOM generation to be enabled by default (mostly) Previously SBOM generation was opt-in. This patch changes the generation of the plain-text tag:value SBOM to be enabled by default, except for: - developer builds - no-prefix builds - standalone tests or examples - cmake build tests The JSON SBOM generation and the verification steps have also been changed to be enabled by default, but only if the Python dependencies can be found. If the dependencies are not found, the build will skip the generation and verification steps. Four new configure options have been added to control these aspects: -(no-)sbom-json: Allows explicitly enabling or disabling JSON SBOM generation -(no-)sbom-json-required: Fails the build if JSON SBOM generation Python dependencies are not found -(no-)sbom-verify: Allows explicitly enabling or disabling SBOM verification -(no-)sbom-verify-required: Fails the build if SBOM verification Python dependencies are not found There are corresponding CMake variables for each of the configure options, see the cmake mapping document. [ChangeLog][Build Systems] SBOM generation is now enabled by default, when building Qt, except for developer builds and no-prefix builds. JSON SBOM generation is enabled by default if the required Python dependencies are available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6dbe1869f8342154a89ff2ab84ac53c9ef1b2eb7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-14 17:05:23 +02:00
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(QT_INTERNAL_SBOM_AUDIT OR QT_INTERNAL_SBOM_AUDIT_NO_ERROR)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options AUDIT)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Change SBOM generation to be enabled by default (mostly) Previously SBOM generation was opt-in. This patch changes the generation of the plain-text tag:value SBOM to be enabled by default, except for: - developer builds - no-prefix builds - standalone tests or examples - cmake build tests The JSON SBOM generation and the verification steps have also been changed to be enabled by default, but only if the Python dependencies can be found. If the dependencies are not found, the build will skip the generation and verification steps. Four new configure options have been added to control these aspects: -(no-)sbom-json: Allows explicitly enabling or disabling JSON SBOM generation -(no-)sbom-json-required: Fails the build if JSON SBOM generation Python dependencies are not found -(no-)sbom-verify: Allows explicitly enabling or disabling SBOM verification -(no-)sbom-verify-required: Fails the build if SBOM verification Python dependencies are not found There are corresponding CMake variables for each of the configure options, see the cmake mapping document. [ChangeLog][Build Systems] SBOM generation is now enabled by default, when building Qt, except for developer builds and no-prefix builds. JSON SBOM generation is enabled by default if the required Python dependencies are available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6dbe1869f8342154a89ff2ab84ac53c9ef1b2eb7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-14 17:05:23 +02:00
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(QT_INTERNAL_SBOM_AUDIT_NO_ERROR)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options AUDIT_NO_ERROR)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
if(QT_GENERATE_SOURCE_SBOM)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options GENERATE_SOURCE_SBOM)
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
endif()
if(QT_LINT_SOURCE_SBOM)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options LINT_SOURCE_SBOM)
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
endif()
if(QT_INTERNAL_LINT_SOURCE_SBOM_NO_ERROR)
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
list(APPEND options LINT_SOURCE_SBOM_NO_ERROR)
CMake: Allow generating and verifying a source SBOM using 'reuse' tool Add code to generate and install a source SBOM SPDX file for every repo. It relies on the python 'reuse' tool being installed and available in PATH. Also add code to allow running 'reuse lint', which checks compliance with the reuse specification. The features are only enabled when configuring with -DQT_GENERATE_SBOM=ON -DQT_GENERATE_SOURCE_SBOM=ON -DQT_LINT_SOURCE_SBOM=ON which will be the case for our CI in a follow up patch. Because most of our repos are not yet reuse compliant, the actual generation of the source SBOM and the linting is skipped if the project root directory does not contain a REUSE.toml file. This allows incremental handling of each repository, while also enforcing the compliance at installation time when the REUSE.toml file is actually there. The source SBOM generation and linting will run at installation time, but they can also be manually triggered at build time using the ninja 'sbom' and 'reuse_lint' custom targets. Various opt outs are provided as a fail safe: - QT_FORCE_SOURCE_SBOM_GENERATION to force source sbom generation even if a REUSE.toml file is not present in the root source dir - QT_FORCE_REUSE_LINT_ERROR to force linting to error out, even if a REUSE.toml file is not present - QT_FORCE_SKIP_REUSE_LINT_ON_INSTALL to skip linting at installation time, but allow running it at build time These can be set either locally or conditionally passed to CMake inside repo-specific Coin instructions. Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-125211 Change-Id: I664e69830936c4427688143ee86b98782c1733ab Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-17 12:43:36 +02:00
endif()
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
_qt_internal_sbom_setup_project_ops_generation(${options})
endfunction()
CMake: Add function to set up SBOM generation defaults Move the setting up of options like QT_GENERATE_SBOM into a separate function, that can be called by projects other than Qt. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I337c4e9b0b3154ff97f494887bf88ee01e8ed00c Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-02-27 16:53:23 +01:00
# Sets up SBOM generation and verification options.
# By default SBOM generation is disabled.
# By default JSON generation and SBOM verification are enabled by default, if the dependencies
# are present, otherwise they will be silently skipped. Unless the user explicitly requests to
# fail the build if the dependencies are not found.
#
# The QT_GENERATE_SBOM_DEFAULT option can be set by a project to change the default value.
function(_qt_internal_setup_sbom)
set(opt_args "")
set(single_args
GENERATE_SBOM_DEFAULT
)
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
set(default_value "OFF")
if(NOT "${arg_GENERATE_SBOM_DEFAULT}" STREQUAL "")
set(default_value "${arg_GENERATE_SBOM_DEFAULT}")
endif()
option(QT_GENERATE_SBOM "Generate SBOM documents in SPDX v2.3 tag:value format."
"${default_value}")
string(CONCAT help_string
"Generate SBOM documents in SPDX v2.3 JSON format if required python dependency "
"spdx-tools is available"
)
option(QT_SBOM_GENERATE_JSON
"${help_string}" ON)
option(QT_SBOM_REQUIRE_GENERATE_JSON
"Error out if JSON SBOM generation depdendency is not found." OFF)
option(QT_SBOM_VERIFY "Verify generated SBOM documents using python spdx-tools package." ON)
option(QT_SBOM_REQUIRE_VERIFY
"Error out if SBOM verification dependencies are not found." OFF)
endfunction()
CMake: Move SBOM project ops setup to run at project begin time Previously Python and other sbom tooling was looked up at SBOM project end time. This was fine for the use cases we had so far, but it prevents introducing new API that wants call the spdx tools before the end of the project. Such API will be introduced in a follow up change. Move the python interpreter, python dependency lookup and other sbom tooling setup to happen at project begin time. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ic8884e378c0ffd9720ede26b0c61f4122f3bb9d9 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-12-09 16:36:58 +01:00
# Ends repo sbom project generation.
# Should be called after all relevant targets are added to the sbom.
# Handles registering sbom info for recorded system libraries and then creates the sbom build
# and install rules.
function(_qt_internal_sbom_end_project)
if(NOT QT_GENERATE_SBOM)
return()
endif()
# Now that we know which system libraries are linked against because we added all
# subdirectories, we can add the recorded system libs to the sbom.
_qt_internal_sbom_add_recorded_system_libraries()
# Run sbom finalization for targets that had it scheduled, but haven't run yet.
# This can happen when _qt_internal_sbom_end_project is called within the same
# subdirectory scope as where the targets are meant to be finalized, but that would be too late
# and the targets wouldn't be added to the sbom.
# This would mostly happen in user projects, and not Qt repos, because in Qt repos we afaik
# never create targets in the root cmakelists (aside from the qtbase Platform targets).
get_cmake_property(targets _qt_internal_sbom_targets_waiting_for_finalization)
if(targets)
foreach(target IN LISTS targets)
_qt_internal_finalize_sbom("${target}")
endforeach()
endif()
_qt_internal_sbom_end_project_generate()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Clean up external document ref properties, because each repo needs to start from scratch
# in a top-level build.
get_cmake_property(known_external_documents _qt_known_external_documents)
set_property(GLOBAL PROPERTY _qt_known_external_documents "")
foreach(external_document IN LISTS known_external_documents)
set_property(GLOBAL PROPERTY _qt_known_external_documents_${external_document} "")
endforeach()
set_property(GLOBAL PROPERTY _qt_internal_sbom_repo_begin_called FALSE)
CMake: Add configure-time dependency on project attribution files Otherwise the SBOM is not regenerated if the attribution files are modified. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I5b3f62e254aa70021ed06fac73f881bcbb110c31 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-09-03 12:56:40 +02:00
# Add configure-time dependency on project attribution files.
get_property(attribution_files GLOBAL PROPERTY _qt_internal_project_attribution_files)
Introduce _qt_internal_append_cmake_configure_depends The function append the unique entries to the CMAKE_CONFIGURE_DEPENDS property. This suppress the ninja recompat issue, which complains about the duplicated entries in that come from the CMAKE_CONFIGURE_DEPENDS property. It's likely the CMake issue, but we may work around it. Pick-to: 6.8 6.9 6.10 Change-Id: I2f10834b0dca3d2aa08fe13fba69849e97fa77d0 Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2025-03-11 13:48:28 +01:00
_qt_internal_append_cmake_configure_depends(${attribution_files})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endfunction()
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
# Automatically begins sbom generation for a qt git repo unless QT_SKIP_SBOM_AUTO_PROJECT is TRUE.
function(_qt_internal_sbom_auto_begin_qt_repo_project)
# Allow skipping auto generation of sbom project, in case it needs to be manually adjusted with
# extra parameters.
if(QT_SKIP_SBOM_AUTO_PROJECT)
return()
endif()
_qt_internal_sbom_begin_qt_repo_project()
endfunction()
# Sets up sbom generation for a qt git repo or qt-git-repo-sub-project (e.g. qtpdf in qtwebengine).
#
# In the case of a qt-git-repo-sub-project, the function expects the following options:
# - SBOM_PROJECT_NAME (e.g. QtPdf)
# - QT_REPO_PROJECT_NAME (e.g. QtWebEngine)
#
# Expects the following variables to always be set before the function call:
# - QT_STAGING_PREFIX
# - INSTALL_SBOMDIR
function(_qt_internal_sbom_begin_qt_repo_project)
set(opt_args "")
set(single_args
SBOM_PROJECT_NAME
QT_REPO_PROJECT_NAME
)
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
set(sbom_project_args "")
_qt_internal_forward_function_args(
FORWARD_APPEND
FORWARD_PREFIX arg
FORWARD_OUT_VAR sbom_project_args
FORWARD_OPTIONS
${opt_args}
FORWARD_SINGLE
${single_args}
FORWARD_MULTI
${multi_args}
)
_qt_internal_sbom_begin_project(
INSTALL_SBOM_DIR "${INSTALL_SBOMDIR}"
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
USE_GIT_VERSION
__QT_INTERNAL_HANDLE_QT_REPO
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
${sbom_project_args}
)
endfunction()
# Automatically ends sbom generation for a qt git repo unless QT_SKIP_SBOM_AUTO_PROJECT is TRUE.
function(_qt_internal_sbom_auto_end_qt_repo_project)
# Allow skipping auto generation of sbom project, in case it needs to be manually adjusted with
# extra parameters.
if(QT_SKIP_SBOM_AUTO_PROJECT)
return()
endif()
_qt_internal_sbom_end_qt_repo_project()
endfunction()
CMake: Fix SBOM inline comment Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I53696003b144ad628169a345743e343cdfc022ae Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 10:23:36 +01:00
# Ends sbom generation for a qt git repo or qt-git-repo-sub-project.
CMake: Support manual multiple projects within-a-repo SBOM generation Certain repositories like qtwebengine contain multiple projects from the perspective of online installer packaging. In this case the QtWebEngine and QtPdf projects are expected to have separate SBOM documents. Introduce a new QT_SKIP_SBOM_AUTO_PROJECT variable that can be set before qt_build_repo to disable the auto-generation of an SBOM document for the current repo project. Introduce two new internal functions qt_internal_sbom_begin/end_qt_repo_project to allow to manually start and end the SBOM generation for a project within a repo. Because the intermediate file names that assemble the SBOM use the project name as a key, and the project name would be the same for qtwebengine, allow differentiating between the current project name and the real qt repo project name. The current project name is used for the file names, whereas the real qt repo project name is used to extract the dependencies on other repos, to ensure correct dependency build rules. As a drive-by, improve the document dir path search list when an SBOM document can't be found. Pick-to: 6.8 Task-number: QTBUG-128893 Task-number: QTBUG-122899 Change-Id: I61b68098242e7c49b98420265c29af78303c3233 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-14 16:43:13 +01:00
function(_qt_internal_sbom_end_qt_repo_project)
_qt_internal_sbom_end_project()
endfunction()
CMake: Add a fake deterministic SBOM generation option Replaces content like timestamps, version strings and checksums with constant values to make the SBOM generation deterministic. This is useful for reproducible builds, to allow cleaner inter-diffs while developing new SBOM features. Can be enabled by configuring with -DQT_SBOM_FAKE_DETERMINISTIC_BUILD=ON Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Id0003bdd23fe57abd70213a2a108885861693242 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 10:05:05 +01:00
# Enables a fake deterministic SBOM build, for easier inter-diffs between sbom files. Useful
# for local development.
function(_qt_internal_sbom_setup_fake_deterministic_build)
if(NOT DEFINED QT_SBOM_FAKE_DETERMINISTIC_BUILD)
return()
endif()
if(QT_SBOM_FAKE_DETERMINISTIC_BUILD)
set(value "ON")
elseif()
set(value "OFF")
endif()
set(QT_SBOM_FAKE_GIT_VERSION "${value}" CACHE BOOL "SBOM fake git version")
set(QT_SBOM_FAKE_TIMESTAMP "${value}" CACHE BOOL "SBOM fake timestamp")
set(QT_SBOM_FAKE_CHECKSUM "${value}" CACHE BOOL "SBOM fake checksums")
endfunction()
CMake: Rework SBOM PURL handling Previously, only 3 purl entries could be added to a target, which tightly coupled to Qt's needs: a QT one pointing to code.qt.io, a MIRROR one pointing to github, and an upstream one pointing to some upstream third party location. Rework the implementation to allow for an unlimited number of PURL entries and to allow more flexibility when adding PURLs in a user project. The new syntax for adding PURLs to a target, which is also the basis for a future public API is as follows: qt_internal_add_sbom(<target> PURLS [[PURL_ENTRY PURL_ID <id> PURL_TYPE <type> PURL_NAMESPACE <namespace> PURL_NAME <name> PURL_VERSION <version>]...] PURL_VALUES [purl-string...] ) The PURLS keyword is used to specify multiple PURL entries, each starting with the PURL_ENTRY keyword. The PURL_VALUES keyword is used to specify a list of pre-built purl strings. PURL_ID is an optional argument used to identify a specific purl entry, which is mostly needed for Qt's needs, to post-process them further. The rest of the options are pre-existing from the previous implementation. Implementation-wise, there's a new custom parser to be able to parse and validate PURL_ENTRY arguments. The VERSION option was renamed to PACKAGE_VERSION, to avoid some issues in cmake_parse_arguments parsing with nested VERSION options. The NO_PURL option was removed because it makes no sense in the new implementation, because if you specify some PURL arguments, there is already an intention to generate a PURL entry. Qt entities no longer have a restriction on which specific purl ids they can have. The new Qt specific purl IDs have been renamed: - QT -> GENERIC - MIRROR -> GITHUB Amends f7e1123620b623be0c321b54eaba7a1d618a7ce1 Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I050decece1c6d9e6e0e06547043f864d6f497ea7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-24 18:13:54 +01:00
# Helper to get purl entry-specific options.
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
macro(_qt_internal_get_sbom_purl_parsing_options opt_args single_args multi_args)
CMake: Improve SBOM PURL handling Tools that consume SBOMs expect that each SBOM package will contain a unique purl: https://github.com/package-url/purl-spec Each Qt target maps to an SBOM package, so generate a target-specific PURL for each Qt target, by using the combination of the repo name and target name as the purl name. The purl external reference will then look something like: ExternalRef: PACKAGE-MANAGER purl pkg:/TheQtCompany/qtbase-Gui@6.8.0 Also allow customizing the purl for each target by specifying one of the following options to functions like qt_internal_add_module or qt_internal_extend_sbom: - PURL_TYPE - PURL_NAMESPACE - PURL_NAME - PURL_VERSION - PURL_SUBPATH - PURL_QUALIFIERS - NO_PURL - NO_DEFAULT_QT_PURL Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6926dd773a0ef6fc688664bcac7b23483ecbabe6 Reviewed-by: Kai Köhne <kai.koehne@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-22 17:26:29 +02:00
set(${opt_args}
NO_DEFAULT_QT_PURL
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
PURL_USE_PACKAGE_VERSION
CMake: Improve SBOM PURL handling Tools that consume SBOMs expect that each SBOM package will contain a unique purl: https://github.com/package-url/purl-spec Each Qt target maps to an SBOM package, so generate a target-specific PURL for each Qt target, by using the combination of the repo name and target name as the purl name. The purl external reference will then look something like: ExternalRef: PACKAGE-MANAGER purl pkg:/TheQtCompany/qtbase-Gui@6.8.0 Also allow customizing the purl for each target by specifying one of the following options to functions like qt_internal_add_module or qt_internal_extend_sbom: - PURL_TYPE - PURL_NAMESPACE - PURL_NAME - PURL_VERSION - PURL_SUBPATH - PURL_QUALIFIERS - NO_PURL - NO_DEFAULT_QT_PURL Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6926dd773a0ef6fc688664bcac7b23483ecbabe6 Reviewed-by: Kai Köhne <kai.koehne@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-22 17:26:29 +02:00
)
set(${single_args}
CMake: Rework SBOM PURL handling Previously, only 3 purl entries could be added to a target, which tightly coupled to Qt's needs: a QT one pointing to code.qt.io, a MIRROR one pointing to github, and an upstream one pointing to some upstream third party location. Rework the implementation to allow for an unlimited number of PURL entries and to allow more flexibility when adding PURLs in a user project. The new syntax for adding PURLs to a target, which is also the basis for a future public API is as follows: qt_internal_add_sbom(<target> PURLS [[PURL_ENTRY PURL_ID <id> PURL_TYPE <type> PURL_NAMESPACE <namespace> PURL_NAME <name> PURL_VERSION <version>]...] PURL_VALUES [purl-string...] ) The PURLS keyword is used to specify multiple PURL entries, each starting with the PURL_ENTRY keyword. The PURL_VALUES keyword is used to specify a list of pre-built purl strings. PURL_ID is an optional argument used to identify a specific purl entry, which is mostly needed for Qt's needs, to post-process them further. The rest of the options are pre-existing from the previous implementation. Implementation-wise, there's a new custom parser to be able to parse and validate PURL_ENTRY arguments. The VERSION option was renamed to PACKAGE_VERSION, to avoid some issues in cmake_parse_arguments parsing with nested VERSION options. The NO_PURL option was removed because it makes no sense in the new implementation, because if you specify some PURL arguments, there is already an intention to generate a PURL entry. Qt entities no longer have a restriction on which specific purl ids they can have. The new Qt specific purl IDs have been renamed: - QT -> GENERIC - MIRROR -> GITHUB Amends f7e1123620b623be0c321b54eaba7a1d618a7ce1 Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I050decece1c6d9e6e0e06547043f864d6f497ea7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-24 18:13:54 +01:00
PURL_ID
CMake: Improve SBOM PURL handling Tools that consume SBOMs expect that each SBOM package will contain a unique purl: https://github.com/package-url/purl-spec Each Qt target maps to an SBOM package, so generate a target-specific PURL for each Qt target, by using the combination of the repo name and target name as the purl name. The purl external reference will then look something like: ExternalRef: PACKAGE-MANAGER purl pkg:/TheQtCompany/qtbase-Gui@6.8.0 Also allow customizing the purl for each target by specifying one of the following options to functions like qt_internal_add_module or qt_internal_extend_sbom: - PURL_TYPE - PURL_NAMESPACE - PURL_NAME - PURL_VERSION - PURL_SUBPATH - PURL_QUALIFIERS - NO_PURL - NO_DEFAULT_QT_PURL Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6926dd773a0ef6fc688664bcac7b23483ecbabe6 Reviewed-by: Kai Köhne <kai.koehne@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-22 17:26:29 +02:00
PURL_TYPE
PURL_NAMESPACE
PURL_NAME
PURL_VERSION
PURL_SUBPATH
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
PURL_VCS_URL
CMake: Improve SBOM PURL handling Tools that consume SBOMs expect that each SBOM package will contain a unique purl: https://github.com/package-url/purl-spec Each Qt target maps to an SBOM package, so generate a target-specific PURL for each Qt target, by using the combination of the repo name and target name as the purl name. The purl external reference will then look something like: ExternalRef: PACKAGE-MANAGER purl pkg:/TheQtCompany/qtbase-Gui@6.8.0 Also allow customizing the purl for each target by specifying one of the following options to functions like qt_internal_add_module or qt_internal_extend_sbom: - PURL_TYPE - PURL_NAMESPACE - PURL_NAME - PURL_VERSION - PURL_SUBPATH - PURL_QUALIFIERS - NO_PURL - NO_DEFAULT_QT_PURL Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6926dd773a0ef6fc688664bcac7b23483ecbabe6 Reviewed-by: Kai Köhne <kai.koehne@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-22 17:26:29 +02:00
)
set(${multi_args}
PURL_QUALIFIERS
)
endmacro()
CMake: Rework SBOM PURL handling Previously, only 3 purl entries could be added to a target, which tightly coupled to Qt's needs: a QT one pointing to code.qt.io, a MIRROR one pointing to github, and an upstream one pointing to some upstream third party location. Rework the implementation to allow for an unlimited number of PURL entries and to allow more flexibility when adding PURLs in a user project. The new syntax for adding PURLs to a target, which is also the basis for a future public API is as follows: qt_internal_add_sbom(<target> PURLS [[PURL_ENTRY PURL_ID <id> PURL_TYPE <type> PURL_NAMESPACE <namespace> PURL_NAME <name> PURL_VERSION <version>]...] PURL_VALUES [purl-string...] ) The PURLS keyword is used to specify multiple PURL entries, each starting with the PURL_ENTRY keyword. The PURL_VALUES keyword is used to specify a list of pre-built purl strings. PURL_ID is an optional argument used to identify a specific purl entry, which is mostly needed for Qt's needs, to post-process them further. The rest of the options are pre-existing from the previous implementation. Implementation-wise, there's a new custom parser to be able to parse and validate PURL_ENTRY arguments. The VERSION option was renamed to PACKAGE_VERSION, to avoid some issues in cmake_parse_arguments parsing with nested VERSION options. The NO_PURL option was removed because it makes no sense in the new implementation, because if you specify some PURL arguments, there is already an intention to generate a PURL entry. Qt entities no longer have a restriction on which specific purl ids they can have. The new Qt specific purl IDs have been renamed: - QT -> GENERIC - MIRROR -> GITHUB Amends f7e1123620b623be0c321b54eaba7a1d618a7ce1 Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I050decece1c6d9e6e0e06547043f864d6f497ea7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-24 18:13:54 +01:00
# Helper to get the purl options that should be recongized by sbom functions like
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
# _qt_internal_sbom_add_target.
macro(_qt_internal_get_sbom_purl_add_target_options opt_args single_args multi_args)
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
set(${opt_args}
__QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_PURL
)
CMake: Handle multi-value PURLs and CPEs in attribution files Qt attribution json files might want to specify more than one PURL or CPE for a given entry. Change the build system code to detect whether a JSON PURL or CPE key contains an array of values, and if so, convert it to a cmake list and use that for further SBOM handling. As a result, the PURL_QT_VALUE, PURL_3RDPARTY_UPSTREAM_VALUE, and PURL_MIRROR_VALUE getting an 'S' at the end, aka they are renamed to PURL_QT_VALUES, PURL_3RDPARTY_UPSTREAM_VALUES, and PURL_MIRROR_VALUES. Also the attribution key is now called just PURL instead of UpstreamPURL. The CPE json attribution key and option name stay the same. Amends 47fd38be4bce0958fcfce8080d1580c4e3c2a15b Amends 95b7fe49900904d19fca21876c84f97c2a6ae03d Amends f7e1123620b623be0c321b54eaba7a1d618a7ce1 Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ieec919901c3b44df80bc196536f68632a9761d92 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-08-22 19:16:33 +02:00
set(${single_args} "")
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
set(${multi_args}
CMake: Rework SBOM PURL handling Previously, only 3 purl entries could be added to a target, which tightly coupled to Qt's needs: a QT one pointing to code.qt.io, a MIRROR one pointing to github, and an upstream one pointing to some upstream third party location. Rework the implementation to allow for an unlimited number of PURL entries and to allow more flexibility when adding PURLs in a user project. The new syntax for adding PURLs to a target, which is also the basis for a future public API is as follows: qt_internal_add_sbom(<target> PURLS [[PURL_ENTRY PURL_ID <id> PURL_TYPE <type> PURL_NAMESPACE <namespace> PURL_NAME <name> PURL_VERSION <version>]...] PURL_VALUES [purl-string...] ) The PURLS keyword is used to specify multiple PURL entries, each starting with the PURL_ENTRY keyword. The PURL_VALUES keyword is used to specify a list of pre-built purl strings. PURL_ID is an optional argument used to identify a specific purl entry, which is mostly needed for Qt's needs, to post-process them further. The rest of the options are pre-existing from the previous implementation. Implementation-wise, there's a new custom parser to be able to parse and validate PURL_ENTRY arguments. The VERSION option was renamed to PACKAGE_VERSION, to avoid some issues in cmake_parse_arguments parsing with nested VERSION options. The NO_PURL option was removed because it makes no sense in the new implementation, because if you specify some PURL arguments, there is already an intention to generate a PURL entry. Qt entities no longer have a restriction on which specific purl ids they can have. The new Qt specific purl IDs have been renamed: - QT -> GENERIC - MIRROR -> GITHUB Amends f7e1123620b623be0c321b54eaba7a1d618a7ce1 Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I050decece1c6d9e6e0e06547043f864d6f497ea7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-24 18:13:54 +01:00
PURLS
PURL_VALUES
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
)
endmacro()
# Helper to get purl options that should be forwarded from _qt_internal_sbom_add_target to
# _qt_internal_sbom_handle_purl_values.
macro(_qt_internal_get_sbom_purl_handling_options opt_args single_args multi_args)
CMake: Rework SBOM PURL handling Previously, only 3 purl entries could be added to a target, which tightly coupled to Qt's needs: a QT one pointing to code.qt.io, a MIRROR one pointing to github, and an upstream one pointing to some upstream third party location. Rework the implementation to allow for an unlimited number of PURL entries and to allow more flexibility when adding PURLs in a user project. The new syntax for adding PURLs to a target, which is also the basis for a future public API is as follows: qt_internal_add_sbom(<target> PURLS [[PURL_ENTRY PURL_ID <id> PURL_TYPE <type> PURL_NAMESPACE <namespace> PURL_NAME <name> PURL_VERSION <version>]...] PURL_VALUES [purl-string...] ) The PURLS keyword is used to specify multiple PURL entries, each starting with the PURL_ENTRY keyword. The PURL_VALUES keyword is used to specify a list of pre-built purl strings. PURL_ID is an optional argument used to identify a specific purl entry, which is mostly needed for Qt's needs, to post-process them further. The rest of the options are pre-existing from the previous implementation. Implementation-wise, there's a new custom parser to be able to parse and validate PURL_ENTRY arguments. The VERSION option was renamed to PACKAGE_VERSION, to avoid some issues in cmake_parse_arguments parsing with nested VERSION options. The NO_PURL option was removed because it makes no sense in the new implementation, because if you specify some PURL arguments, there is already an intention to generate a PURL entry. Qt entities no longer have a restriction on which specific purl ids they can have. The new Qt specific purl IDs have been renamed: - QT -> GENERIC - MIRROR -> GITHUB Amends f7e1123620b623be0c321b54eaba7a1d618a7ce1 Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I050decece1c6d9e6e0e06547043f864d6f497ea7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-24 18:13:54 +01:00
set(${opt_args} "")
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
set(${single_args}
SUPPLIER
TYPE
CMake: Rework SBOM PURL handling Previously, only 3 purl entries could be added to a target, which tightly coupled to Qt's needs: a QT one pointing to code.qt.io, a MIRROR one pointing to github, and an upstream one pointing to some upstream third party location. Rework the implementation to allow for an unlimited number of PURL entries and to allow more flexibility when adding PURLs in a user project. The new syntax for adding PURLs to a target, which is also the basis for a future public API is as follows: qt_internal_add_sbom(<target> PURLS [[PURL_ENTRY PURL_ID <id> PURL_TYPE <type> PURL_NAMESPACE <namespace> PURL_NAME <name> PURL_VERSION <version>]...] PURL_VALUES [purl-string...] ) The PURLS keyword is used to specify multiple PURL entries, each starting with the PURL_ENTRY keyword. The PURL_VALUES keyword is used to specify a list of pre-built purl strings. PURL_ID is an optional argument used to identify a specific purl entry, which is mostly needed for Qt's needs, to post-process them further. The rest of the options are pre-existing from the previous implementation. Implementation-wise, there's a new custom parser to be able to parse and validate PURL_ENTRY arguments. The VERSION option was renamed to PACKAGE_VERSION, to avoid some issues in cmake_parse_arguments parsing with nested VERSION options. The NO_PURL option was removed because it makes no sense in the new implementation, because if you specify some PURL arguments, there is already an intention to generate a PURL entry. Qt entities no longer have a restriction on which specific purl ids they can have. The new Qt specific purl IDs have been renamed: - QT -> GENERIC - MIRROR -> GITHUB Amends f7e1123620b623be0c321b54eaba7a1d618a7ce1 Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I050decece1c6d9e6e0e06547043f864d6f497ea7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-24 18:13:54 +01:00
PACKAGE_VERSION
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
)
set(${multi_args} "")
_qt_internal_get_sbom_purl_add_target_options(
purl_add_target_opt_args purl_add_target_single_args purl_add_target_multi_args)
list(APPEND ${opt_args} ${purl_add_target_opt_args})
list(APPEND ${single_args} ${purl_add_target_single_args})
list(APPEND ${multi_args} ${purl_add_target_multi_args})
endmacro()
CMake: Allow specifying most SBOM options to qt_internal_add_foo Previously, only a very short subset of options related to attribution files could be specified to qt_internal_add_module / qt_internal_extend_target. It is more convenient to allow specifying most (safe) options, instead of calling another function. Unsafe are considered paths like INSTALL_PATH and derivatives, TYPE which is too generic, and some other ones like LIBRARIES which would be duplicated, and causes warnings in cmake_parse_arguments if duplicated. Change the code to allow specifying most SBOM options and forwarding them to _qt_internal_extend_sbom. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6eb723e165edf59973d83c66eace43acdce237de Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-14 19:07:16 +02:00
# Helper to get the options that _qt_internal_sbom_add_target understands, but that are also
# a safe subset for qt_internal_add_module, qt_internal_extend_target, etc to understand.
macro(_qt_internal_get_sbom_add_target_common_options opt_args single_args multi_args)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(${opt_args}
NO_CURRENT_DIR_ATTRIBUTION
CMake: Improve license handling for SBOMs Add a NO_ATTRIBUTION_LICENSE_ID option to allow to skip including license references from qt_attribution.json files. Add a QT_LICENSE_ID option that can take an "id" value and return one of the common license expressions we use throughout Qt repositories: - GPL3 with exception for tools / apps - GFDL for documentation - GPL3 only modules etc. Add a QT_SBOM_DEFAULT_QT_LICENSE_ID_LIBRARIES variable that can be set at directory scope (or root repo scope) to use a specific license id for all qt modules / plugins created within that repo, e.g. the Commercial + GPL3 variant. These can be opted out, to fallback to the "default" license by passing NO_DEFAULT_QT_LICENSE_ID_LIBRARIES. Add a similar variable QT_SBOM_DEFAULT_QT_LICENSE_ID_EXECUTABLES for tools / apps, including a NO_DEFAULT_QT_LICENSE_ID_EXECUTABLES option. Modify the logic for setting licenses for qt targets: - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_EXECUTABLES for tools and apps - or use the 'GPL3 with exception' variant for tools and apps - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_LIBARRIES for modules ands plugins - or use the default Commercial + LGPL3 + GPL variant for modules and plugins Amends 37a5e001277db9e1392a242171ab2b88cb6c3049 Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3a8abac62b9f4b342f91ef139064884f02aa935e Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-06-14 17:51:12 +02:00
NO_ATTRIBUTION_LICENSE_ID
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
NO_DEFAULT_QT_LICENSE
CMake: Improve license handling for SBOMs Add a NO_ATTRIBUTION_LICENSE_ID option to allow to skip including license references from qt_attribution.json files. Add a QT_LICENSE_ID option that can take an "id" value and return one of the common license expressions we use throughout Qt repositories: - GPL3 with exception for tools / apps - GFDL for documentation - GPL3 only modules etc. Add a QT_SBOM_DEFAULT_QT_LICENSE_ID_LIBRARIES variable that can be set at directory scope (or root repo scope) to use a specific license id for all qt modules / plugins created within that repo, e.g. the Commercial + GPL3 variant. These can be opted out, to fallback to the "default" license by passing NO_DEFAULT_QT_LICENSE_ID_LIBRARIES. Add a similar variable QT_SBOM_DEFAULT_QT_LICENSE_ID_EXECUTABLES for tools / apps, including a NO_DEFAULT_QT_LICENSE_ID_EXECUTABLES option. Modify the logic for setting licenses for qt targets: - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_EXECUTABLES for tools and apps - or use the 'GPL3 with exception' variant for tools and apps - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_LIBARRIES for modules ands plugins - or use the default Commercial + LGPL3 + GPL variant for modules and plugins Amends 37a5e001277db9e1392a242171ab2b88cb6c3049 Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3a8abac62b9f4b342f91ef139064884f02aa935e Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-06-14 17:51:12 +02:00
NO_DEFAULT_QT_LICENSE_ID_LIBRARIES
NO_DEFAULT_QT_LICENSE_ID_EXECUTABLES
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
NO_DEFAULT_DIRECTORY_QT_LICENSE
NO_DEFAULT_QT_COPYRIGHTS
NO_DEFAULT_QT_PACKAGE_VERSION
NO_DEFAULT_QT_SUPPLIER
CMake: Allow marking incomplete 3rd party deps for a target's SBOM For some targets, it might not be possible to specify all 3rd party dependencies information for SBOM generation. qtwebengine is one of these cases, where the 3rd party targets are only known to GN, and not CMake. Add a new SBOM_INCOMPLETE_3RD_PARTY_DEPENDENCIES option which can be passed to qt_internal_add_module and friends. This will include an informational message into the SBOM package comment field that the dependency information might be incomplete. Also add an SBOM_PACKAGE_COMMENT option, which can be used to provide further information if necessary. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I5e893e1e205aae4a5591a457be88d5db54fa8fc3 Reviewed-by: Kai Köhne <kai.koehne@qt.io>
2024-07-17 17:25:12 +02:00
SBOM_INCOMPLETE_3RD_PARTY_DEPENDENCIES
CMake: Detect 3rd party header modules during SBOM generation They should inherit the version of the 3rd party library they are part of, and not be treated as Qt modules. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ibf99f4481fbc1acca488fc96cca048298b080d35 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-08-02 15:06:31 +02:00
IS_QT_3RD_PARTY_HEADER_MODULE
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
USE_ATTRIBUTION_FILES
CMake: Allow creating nested SBOM attribution targets for all entities Allow creating nested SBOM attribution targets also for non-Qt targets in user projects. This will allow using qt_attribution.json files in user projects. Rename PARENT_TARGET to ATTRIBUTION_PARENT_TARGET to make it clear what the parent target is for. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia08ee934a9dc03827fcc26d0fc90e072499e8a21 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-02-24 16:42:00 +01:00
CREATE_SBOM_FOR_EACH_ATTRIBUTION
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
__QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_PACKAGE_VERSION
__QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_SUPPLIER
__QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_DOWNLOAD_LOCATION
__QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_LICENSE
__QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_COPYRIGHTS
__QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_CPE
__QT_INTERNAL_HANDLE_QT_ENTITY_ATTRIBUTION_FILES
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
set(${single_args}
PACKAGE_VERSION
FRIENDLY_PACKAGE_NAME
CMake: Add SUPPLIER as an option to SBOM packages Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I22cae71dcc7582126320e17271a8c55bea1ad393 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 17:22:04 +01:00
SUPPLIER
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CPE_VENDOR
CPE_PRODUCT
LICENSE_EXPRESSION
CMake: Improve license handling for SBOMs Add a NO_ATTRIBUTION_LICENSE_ID option to allow to skip including license references from qt_attribution.json files. Add a QT_LICENSE_ID option that can take an "id" value and return one of the common license expressions we use throughout Qt repositories: - GPL3 with exception for tools / apps - GFDL for documentation - GPL3 only modules etc. Add a QT_SBOM_DEFAULT_QT_LICENSE_ID_LIBRARIES variable that can be set at directory scope (or root repo scope) to use a specific license id for all qt modules / plugins created within that repo, e.g. the Commercial + GPL3 variant. These can be opted out, to fallback to the "default" license by passing NO_DEFAULT_QT_LICENSE_ID_LIBRARIES. Add a similar variable QT_SBOM_DEFAULT_QT_LICENSE_ID_EXECUTABLES for tools / apps, including a NO_DEFAULT_QT_LICENSE_ID_EXECUTABLES option. Modify the logic for setting licenses for qt targets: - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_EXECUTABLES for tools and apps - or use the 'GPL3 with exception' variant for tools and apps - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_LIBARRIES for modules ands plugins - or use the default Commercial + LGPL3 + GPL variant for modules and plugins Amends 37a5e001277db9e1392a242171ab2b88cb6c3049 Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3a8abac62b9f4b342f91ef139064884f02aa935e Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-06-14 17:51:12 +02:00
QT_LICENSE_ID
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
DOWNLOAD_LOCATION
ATTRIBUTION_ENTRY_INDEX
CMake: Allow creating nested SBOM attribution targets for all entities Allow creating nested SBOM attribution targets also for non-Qt targets in user projects. This will allow using qt_attribution.json files in user projects. Rename PARENT_TARGET to ATTRIBUTION_PARENT_TARGET to make it clear what the parent target is for. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia08ee934a9dc03827fcc26d0fc90e072499e8a21 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-02-24 16:42:00 +01:00
ATTRIBUTION_PARENT_TARGET
CMake: Allow marking incomplete 3rd party deps for a target's SBOM For some targets, it might not be possible to specify all 3rd party dependencies information for SBOM generation. qtwebengine is one of these cases, where the 3rd party targets are only known to GN, and not CMake. Add a new SBOM_INCOMPLETE_3RD_PARTY_DEPENDENCIES option which can be passed to qt_internal_add_module and friends. This will include an informational message into the SBOM package comment field that the dependency information might be incomplete. Also add an SBOM_PACKAGE_COMMENT option, which can be used to provide further information if necessary. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I5e893e1e205aae4a5591a457be88d5db54fa8fc3 Reviewed-by: Kai Köhne <kai.koehne@qt.io>
2024-07-17 17:25:12 +02:00
SBOM_PACKAGE_COMMENT
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
set(${multi_args}
COPYRIGHTS
CPE
SBOM_DEPENDENCIES
ATTRIBUTION_FILE_PATHS
ATTRIBUTION_FILE_DIR_PATHS
CMake: Allow including specific attribution ids in SBOMs Previously one could attach all attributions from a qt_attribution.json file to a target, or a specific entry based on a given numeric index. This is not always practical, as the index of an entry can change, and including all entries might be too much. Allow specifying a list of attribution ids to include in the SBOM. This will parse all the attribution entries and only include those that match the given ids. If a list of ids is provided, and any of them is not found, an error will be shown. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I2ca442fe0cd6d18773861f90f32ab7380aaeacdd Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-02-24 14:52:12 +01:00
ATTRIBUTION_IDS
CMake: Allow adding custom SBOM relationships to targets and projects Add a new SBOM_RELATIONSHIPS option to qt_internal_extend_target and friends that allows adding custom relationships to the current target. Add a new function qt_internal_sbom_add_project_relationship that allows adding custom relationships to the current project SBOM document. A sample usage might be: qt_internal_sbom_get_project_spdx_id(project_spdx_id) qt_internal_sbom_get_target_spdx_id(Svg svg_spdx_id) qt_internal_extend_target(Svg SBOM_RELATIONSHIPS "${svg_spdx_id} DESCENDANT_OF ${project_spdx_id}" ) qt_internal_sbom_add_project_relationship( RELATIONSHIPS "${svg_spdx_id} CONTAINS NOASSERTION" "${svg_spdx_id} DESCRIBES NOASSERTION" "${project_spdx_id} DESCRIBES NOASSERTION" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ie0119ca71b047c7515e1deaf84a5a67ea01b5274 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 19:13:29 +01:00
SBOM_RELATIONSHIPS
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
CMake: Improve SBOM PURL handling Tools that consume SBOMs expect that each SBOM package will contain a unique purl: https://github.com/package-url/purl-spec Each Qt target maps to an SBOM package, so generate a target-specific PURL for each Qt target, by using the combination of the repo name and target name as the purl name. The purl external reference will then look something like: ExternalRef: PACKAGE-MANAGER purl pkg:/TheQtCompany/qtbase-Gui@6.8.0 Also allow customizing the purl for each target by specifying one of the following options to functions like qt_internal_add_module or qt_internal_extend_sbom: - PURL_TYPE - PURL_NAMESPACE - PURL_NAME - PURL_VERSION - PURL_SUBPATH - PURL_QUALIFIERS - NO_PURL - NO_DEFAULT_QT_PURL Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6926dd773a0ef6fc688664bcac7b23483ecbabe6 Reviewed-by: Kai Köhne <kai.koehne@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-22 17:26:29 +02:00
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
_qt_internal_get_sbom_purl_add_target_options(
purl_add_target_opt_args purl_add_target_single_args purl_add_target_multi_args)
list(APPEND ${opt_args} ${purl_add_target_opt_args})
list(APPEND ${single_args} ${purl_add_target_single_args})
list(APPEND ${multi_args} ${purl_add_target_multi_args})
CMake: Allow specifying most SBOM options to qt_internal_add_foo Previously, only a very short subset of options related to attribution files could be specified to qt_internal_add_module / qt_internal_extend_target. It is more convenient to allow specifying most (safe) options, instead of calling another function. Unsafe are considered paths like INSTALL_PATH and derivatives, TYPE which is too generic, and some other ones like LIBRARIES which would be duplicated, and causes warnings in cmake_parse_arguments if duplicated. Change the code to allow specifying most SBOM options and forwarding them to _qt_internal_extend_sbom. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6eb723e165edf59973d83c66eace43acdce237de Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-14 19:07:16 +02:00
endmacro()
CMake: Propagate SBOM options to auto-created attribution targets Previously if an option like CPE was passed to a 3rd party target that would create nested attribution targets, one per entry in the attribution file, the CPE option would only apply to the first / parent target, but not to the nested targets. This change will now propagate all SBOM options to the nested targets, but only if the parent target is not a Qt entity type like a module, plugin, etc, which means it will only apply to 3rd party attribution targets. The restriction is there because because mostly only 3rd party attribution targets should inherit the same set of CPEs and similar SBOM values. If the restriction proves to be too strict, it will be re-assessed in a future change. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I96cfa046ad611c10877b7a06504b35615b539bbe Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-07-24 17:29:11 +02:00
# Helper to get all known SBOM specific options, without the ones that qt_internal_add_module
# and similar functions understand, like LIBRARIES, INCLUDES, etc.
macro(_qt_internal_get_sbom_specific_options opt_args single_args multi_args)
set(${opt_args} "")
set(${single_args} "")
set(${multi_args} "")
_qt_internal_get_sbom_add_target_common_options(
common_opt_args common_single_args common_multi_args)
list(APPEND ${opt_args} ${common_opt_args})
list(APPEND ${single_args} ${common_single_args})
list(APPEND ${multi_args} ${common_multi_args})
_qt_internal_sbom_get_multi_config_single_args(multi_config_single_args)
list(APPEND ${single_args} ${multi_config_single_args})
endmacro()
CMake: Allow specifying most SBOM options to qt_internal_add_foo Previously, only a very short subset of options related to attribution files could be specified to qt_internal_add_module / qt_internal_extend_target. It is more convenient to allow specifying most (safe) options, instead of calling another function. Unsafe are considered paths like INSTALL_PATH and derivatives, TYPE which is too generic, and some other ones like LIBRARIES which would be duplicated, and causes warnings in cmake_parse_arguments if duplicated. Change the code to allow specifying most SBOM options and forwarding them to _qt_internal_extend_sbom. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6eb723e165edf59973d83c66eace43acdce237de Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-14 19:07:16 +02:00
# Helper to get the options that _qt_internal_sbom_add_target understands.
# Also used in qt_find_package_extend_sbom.
macro(_qt_internal_get_sbom_add_target_options opt_args single_args multi_args)
set(${opt_args}
NO_INSTALL
)
set(${single_args}
TYPE
)
set(${multi_args}
LIBRARIES
PUBLIC_LIBRARIES
)
CMake: Propagate SBOM options to auto-created attribution targets Previously if an option like CPE was passed to a 3rd party target that would create nested attribution targets, one per entry in the attribution file, the CPE option would only apply to the first / parent target, but not to the nested targets. This change will now propagate all SBOM options to the nested targets, but only if the parent target is not a Qt entity type like a module, plugin, etc, which means it will only apply to 3rd party attribution targets. The restriction is there because because mostly only 3rd party attribution targets should inherit the same set of CPEs and similar SBOM values. If the restriction proves to be too strict, it will be re-assessed in a future change. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I96cfa046ad611c10877b7a06504b35615b539bbe Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-07-24 17:29:11 +02:00
_qt_internal_get_sbom_specific_options(
specific_opt_args specific_single_args specific_multi_args)
list(APPEND ${opt_args} ${specific_opt_args})
list(APPEND ${single_args} ${specific_single_args})
list(APPEND ${multi_args} ${specific_multi_args})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endmacro()
# Generate sbom information for a given target.
# Creates:
# - a SPDX package for the target
# - zero or more SPDX file entries for each installed binary file
# - each binary file entry gets a list of 'generated from source files' section
# - dependency relationships to other target packages
# - other relevant information like licenses, copyright, etc.
# For licenses, copyrights, these can either be passed as options, or read from qt_attribution.json
# files.
# For dependencies, these are either specified via options, or read from properties set on the
# target by qt_internal_extend_target.
function(_qt_internal_sbom_add_target target)
if(NOT QT_GENERATE_SBOM)
return()
endif()
_qt_internal_get_sbom_add_target_options(opt_args single_args multi_args)
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
get_target_property(target_type ${target} TYPE)
# Mark the target as a Qt module for sbom processing purposes.
# Needed for non-standard targets like Bootstrap and QtLibraryInfo, that don't have a Qt::
# namespace prefix.
if(arg_TYPE STREQUAL QT_MODULE)
set_target_properties(${target} PROPERTIES _qt_sbom_is_qt_module TRUE)
endif()
set(project_package_options "")
if(arg_FRIENDLY_PACKAGE_NAME)
set(package_name_for_spdx_id "${arg_FRIENDLY_PACKAGE_NAME}")
else()
set(package_name_for_spdx_id "${target}")
endif()
set(package_comment "")
CMake: Allow marking incomplete 3rd party deps for a target's SBOM For some targets, it might not be possible to specify all 3rd party dependencies information for SBOM generation. qtwebengine is one of these cases, where the 3rd party targets are only known to GN, and not CMake. Add a new SBOM_INCOMPLETE_3RD_PARTY_DEPENDENCIES option which can be passed to qt_internal_add_module and friends. This will include an informational message into the SBOM package comment field that the dependency information might be incomplete. Also add an SBOM_PACKAGE_COMMENT option, which can be used to provide further information if necessary. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I5e893e1e205aae4a5591a457be88d5db54fa8fc3 Reviewed-by: Kai Köhne <kai.koehne@qt.io>
2024-07-17 17:25:12 +02:00
if(arg_SBOM_INCOMPLETE_3RD_PARTY_DEPENDENCIES)
string(APPEND package_comment
"Note: This package was marked as not listing all of its consumed 3rd party "
"dependencies.\nThus the licensing and copyright information might be incomplete.\n")
endif()
if(arg_SBOM_PACKAGE_COMMENT)
string(APPEND package_comment "${arg_SBOM_PACKAGE_COMMENT}\n")
endif()
CMake: Add qt exported target name and package name into SBOM Add them to the PackageComment field of the SBOM. Can be useful to map back the SPDX Package to the CMake target or package. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Icbbb90132038c489b55c4ee7a038aea593ce1ff6 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 18:32:07 +01:00
string(APPEND package_comment "CMake target name: ${target}\n")
get_target_property(qt_package_name "${target}" _qt_package_name)
if(qt_package_name)
get_target_property(qt_module_interface_name "${target}" _qt_module_interface_name)
set(namespaced_target_name "${QT_CMAKE_EXPORT_NAMESPACE}::${qt_module_interface_name}")
string(APPEND package_comment
"CMake exported target name: ${namespaced_target_name}\n")
string(APPEND package_comment "Contained in CMake package: ${qt_package_name}\n")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Record the target spdx id right now, so we can refer to it in later attribution targets
# if needed.
_qt_internal_sbom_record_target_spdx_id(${target}
TYPE "${arg_TYPE}"
PACKAGE_NAME "${package_name_for_spdx_id}"
OUT_VAR package_spdx_id
)
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg_USE_ATTRIBUTION_FILES)
set(attribution_args
CMake: Allow creating nested SBOM attribution targets for all entities Allow creating nested SBOM attribution targets also for non-Qt targets in user projects. This will allow using qt_attribution.json files in user projects. Rename PARENT_TARGET to ATTRIBUTION_PARENT_TARGET to make it clear what the parent target is for. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia08ee934a9dc03827fcc26d0fc90e072499e8a21 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-02-24 16:42:00 +01:00
ATTRIBUTION_PARENT_TARGET "${target}"
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
# Forward the sbom specific options when handling attribution files because those might
# create other sbom targets that need to inherit the parent ones.
_qt_internal_get_sbom_specific_options(sbom_opt_args sbom_single_args sbom_multi_args)
_qt_internal_forward_function_args(
FORWARD_APPEND
FORWARD_PREFIX arg
FORWARD_OUT_VAR attribution_args
FORWARD_OPTIONS
${sbom_opt_args}
FORWARD_SINGLE
${sbom_single_args}
FORWARD_MULTI
${sbom_multi_args}
)
CMake: Propagate SBOM options to auto-created attribution targets Previously if an option like CPE was passed to a 3rd party target that would create nested attribution targets, one per entry in the attribution file, the CPE option would only apply to the first / parent target, but not to the nested targets. This change will now propagate all SBOM options to the nested targets, but only if the parent target is not a Qt entity type like a module, plugin, etc, which means it will only apply to 3rd party attribution targets. The restriction is there because because mostly only 3rd party attribution targets should inherit the same set of CPEs and similar SBOM values. If the restriction proves to be too strict, it will be re-assessed in a future change. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I96cfa046ad611c10877b7a06504b35615b539bbe Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-07-24 17:29:11 +02:00
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(NOT arg_NO_CURRENT_DIR_ATTRIBUTION
AND EXISTS "${CMAKE_CURRENT_SOURCE_DIR}/qt_attribution.json")
list(APPEND attribution_args
ATTRIBUTION_FILE_PATHS "${CMAKE_CURRENT_SOURCE_DIR}/qt_attribution.json"
)
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
_qt_internal_sbom_handle_qt_attribution_files(qa ${attribution_args})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Improve license handling for SBOMs Add a NO_ATTRIBUTION_LICENSE_ID option to allow to skip including license references from qt_attribution.json files. Add a QT_LICENSE_ID option that can take an "id" value and return one of the common license expressions we use throughout Qt repositories: - GPL3 with exception for tools / apps - GFDL for documentation - GPL3 only modules etc. Add a QT_SBOM_DEFAULT_QT_LICENSE_ID_LIBRARIES variable that can be set at directory scope (or root repo scope) to use a specific license id for all qt modules / plugins created within that repo, e.g. the Commercial + GPL3 variant. These can be opted out, to fallback to the "default" license by passing NO_DEFAULT_QT_LICENSE_ID_LIBRARIES. Add a similar variable QT_SBOM_DEFAULT_QT_LICENSE_ID_EXECUTABLES for tools / apps, including a NO_DEFAULT_QT_LICENSE_ID_EXECUTABLES option. Modify the logic for setting licenses for qt targets: - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_EXECUTABLES for tools and apps - or use the 'GPL3 with exception' variant for tools and apps - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_LIBARRIES for modules ands plugins - or use the default Commercial + LGPL3 + GPL variant for modules and plugins Amends 37a5e001277db9e1392a242171ab2b88cb6c3049 Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3a8abac62b9f4b342f91ef139064884f02aa935e Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-06-14 17:51:12 +02:00
# Collect license expressions, but in most cases, each expression needs to be abided, so we
# AND the accumulated license expressions.
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(license_expression "")
if(arg_LICENSE_EXPRESSION)
set(license_expression "${arg_LICENSE_EXPRESSION}")
endif()
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg___QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_LICENSE)
_qt_internal_sbom_forward_sbom_add_target_options(sbom_add_target_args)
_qt_internal_sbom_handle_qt_entity_license_expression(${target} ${sbom_add_target_args}
OUT_VAR qt_entity_license_expression)
if(qt_entity_license_expression)
_qt_internal_sbom_join_two_license_ids_with_op(
"${license_expression}" "AND" "${qt_entity_license_expression}"
license_expression)
CMake: Improve license handling for SBOMs Add a NO_ATTRIBUTION_LICENSE_ID option to allow to skip including license references from qt_attribution.json files. Add a QT_LICENSE_ID option that can take an "id" value and return one of the common license expressions we use throughout Qt repositories: - GPL3 with exception for tools / apps - GFDL for documentation - GPL3 only modules etc. Add a QT_SBOM_DEFAULT_QT_LICENSE_ID_LIBRARIES variable that can be set at directory scope (or root repo scope) to use a specific license id for all qt modules / plugins created within that repo, e.g. the Commercial + GPL3 variant. These can be opted out, to fallback to the "default" license by passing NO_DEFAULT_QT_LICENSE_ID_LIBRARIES. Add a similar variable QT_SBOM_DEFAULT_QT_LICENSE_ID_EXECUTABLES for tools / apps, including a NO_DEFAULT_QT_LICENSE_ID_EXECUTABLES option. Modify the logic for setting licenses for qt targets: - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_EXECUTABLES for tools and apps - or use the 'GPL3 with exception' variant for tools and apps - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_LIBARRIES for modules ands plugins - or use the default Commercial + LGPL3 + GPL variant for modules and plugins Amends 37a5e001277db9e1392a242171ab2b88cb6c3049 Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3a8abac62b9f4b342f91ef139064884f02aa935e Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-06-14 17:51:12 +02:00
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Improve license handling for SBOMs Add a NO_ATTRIBUTION_LICENSE_ID option to allow to skip including license references from qt_attribution.json files. Add a QT_LICENSE_ID option that can take an "id" value and return one of the common license expressions we use throughout Qt repositories: - GPL3 with exception for tools / apps - GFDL for documentation - GPL3 only modules etc. Add a QT_SBOM_DEFAULT_QT_LICENSE_ID_LIBRARIES variable that can be set at directory scope (or root repo scope) to use a specific license id for all qt modules / plugins created within that repo, e.g. the Commercial + GPL3 variant. These can be opted out, to fallback to the "default" license by passing NO_DEFAULT_QT_LICENSE_ID_LIBRARIES. Add a similar variable QT_SBOM_DEFAULT_QT_LICENSE_ID_EXECUTABLES for tools / apps, including a NO_DEFAULT_QT_LICENSE_ID_EXECUTABLES option. Modify the logic for setting licenses for qt targets: - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_EXECUTABLES for tools and apps - or use the 'GPL3 with exception' variant for tools and apps - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_LIBARRIES for modules ands plugins - or use the default Commercial + LGPL3 + GPL variant for modules and plugins Amends 37a5e001277db9e1392a242171ab2b88cb6c3049 Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3a8abac62b9f4b342f91ef139064884f02aa935e Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-06-14 17:51:12 +02:00
# Read a license expression from the attribution json file.
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg_USE_ATTRIBUTION_FILES
AND qa_license_id
AND NOT arg_NO_ATTRIBUTION_LICENSE_ID)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(NOT qa_license_id MATCHES "urn:dje:license")
_qt_internal_sbom_join_two_license_ids_with_op(
"${license_expression}" "AND" "${qa_license_id}"
license_expression)
else()
message(DEBUG
"Attribution license id contains invalid spdx license reference: ${qa_license_id}")
set(invalid_license_comment
" Attribution license ID with invalid spdx license reference: ")
string(APPEND invalid_license_comment "${qa_license_id}\n")
string(APPEND package_comment "${invalid_license_comment}")
endif()
endif()
if(license_expression)
list(APPEND project_package_options LICENSE_CONCLUDED "${license_expression}")
CMake: Extract Qt-specific license concluded handling Extract the handling of the license concluded expression for Qt entities into the Qt specific handling file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ib6f9a64f66dae0c971c43e8fffbc54362487b271 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-02-24 19:02:16 +01:00
endif()
CMake: Improve license handling for SBOMs Add a NO_ATTRIBUTION_LICENSE_ID option to allow to skip including license references from qt_attribution.json files. Add a QT_LICENSE_ID option that can take an "id" value and return one of the common license expressions we use throughout Qt repositories: - GPL3 with exception for tools / apps - GFDL for documentation - GPL3 only modules etc. Add a QT_SBOM_DEFAULT_QT_LICENSE_ID_LIBRARIES variable that can be set at directory scope (or root repo scope) to use a specific license id for all qt modules / plugins created within that repo, e.g. the Commercial + GPL3 variant. These can be opted out, to fallback to the "default" license by passing NO_DEFAULT_QT_LICENSE_ID_LIBRARIES. Add a similar variable QT_SBOM_DEFAULT_QT_LICENSE_ID_EXECUTABLES for tools / apps, including a NO_DEFAULT_QT_LICENSE_ID_EXECUTABLES option. Modify the logic for setting licenses for qt targets: - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_EXECUTABLES for tools and apps - or use the 'GPL3 with exception' variant for tools and apps - use whatever is set in QT_SBOM_DEFAULT_QT_LICENSE_LIBARRIES for modules ands plugins - or use the default Commercial + LGPL3 + GPL variant for modules and plugins Amends 37a5e001277db9e1392a242171ab2b88cb6c3049 Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3a8abac62b9f4b342f91ef139064884f02aa935e Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-06-14 17:51:12 +02:00
CMake: Extract Qt-specific license concluded handling Extract the handling of the license concluded expression for Qt entities into the Qt specific handling file. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ib6f9a64f66dae0c971c43e8fffbc54362487b271 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-02-24 19:02:16 +01:00
if(license_expression AND
arg___QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_LICENSE)
_qt_internal_sbom_forward_sbom_add_target_options(sbom_add_target_args)
_qt_internal_sbom_handle_qt_entity_license_declared_expression(${target}
${sbom_add_target_args}
LICENSE_CONCLUDED_EXPRESSION "${license_expression}"
OUT_VAR qt_entity_license_declared_expression)
if(qt_entity_license_declared_expression)
list(APPEND project_package_options
LICENSE_DECLARED "${qt_entity_license_declared_expression}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
endif()
# Copyrights are additive, so we collect them from all sources that were found.
set(copyrights "")
if(arg_COPYRIGHTS)
list(APPEND copyrights "${arg_COPYRIGHTS}")
endif()
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg___QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_COPYRIGHTS)
_qt_internal_sbom_forward_sbom_add_target_options(sbom_add_target_args)
_qt_internal_sbom_handle_qt_entity_copyrights(${target} ${sbom_add_target_args}
OUT_VAR qt_copyrights)
if(qt_copyrights)
list(APPEND copyrights ${qt_copyrights})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
endif()
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg_USE_ATTRIBUTION_FILES AND qa_copyrights)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
list(APPEND copyrights "${qa_copyrights}")
endif()
if(copyrights)
list(JOIN copyrights "\n" copyrights)
list(APPEND project_package_options COPYRIGHT "<text>${copyrights}</text>")
endif()
set(package_version "")
if(arg_PACKAGE_VERSION)
set(package_version "${arg_PACKAGE_VERSION}")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
elseif(arg_USE_ATTRIBUTION_FILES AND qa_version)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(package_version "${qa_version}")
endif()
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg___QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_PACKAGE_VERSION)
_qt_internal_sbom_forward_sbom_add_target_options(sbom_add_target_args)
_qt_internal_sbom_handle_qt_entity_package_version(${target} ${sbom_add_target_args}
OUT_VAR qt_entity_package_version)
if(qt_entity_package_version)
set(package_version "${qt_entity_package_version}")
endif()
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(package_version)
list(APPEND project_package_options VERSION "${package_version}")
endif()
set(supplier "")
CMake: Add SUPPLIER as an option to SBOM packages Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I22cae71dcc7582126320e17271a8c55bea1ad393 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-10 17:22:04 +01:00
if(arg_SUPPLIER)
set(supplier "${arg_SUPPLIER}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg___QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_SUPPLIER)
_qt_internal_sbom_forward_sbom_add_target_options(sbom_add_target_args)
_qt_internal_sbom_handle_qt_entity_supplier(${target} ${sbom_add_target_args}
OUT_VAR qt_entity_supplier)
if(qt_entity_supplier)
set(supplier "${qt_entity_supplier}")
endif()
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(supplier)
list(APPEND project_package_options SUPPLIER "Organization: ${supplier}")
endif()
set(download_location "")
if(arg_DOWNLOAD_LOCATION)
set(download_location "${arg_DOWNLOAD_LOCATION}")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
endif()
if(arg___QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_DOWNLOAD_LOCATION)
_qt_internal_sbom_forward_sbom_add_target_options(sbom_add_target_args)
_qt_internal_sbom_handle_qt_entity_download_location(${target} ${sbom_add_target_args}
OUT_VAR qt_entity_download_location)
if(qt_entity_download_location)
set(download_location "${qt_entity_download_location}")
endif()
endif()
CMake: Remove type restriction on attribution download location So it can be read / used in other non-Qt projects. We don't have such a restriction on other keys read from the qt attribution files. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I225d83b26c68323761a8b13b3e6442057b5f7c7f Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-02-20 19:46:27 +01:00
if(arg_USE_ATTRIBUTION_FILES)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(qa_download_location)
set(download_location "${qa_download_location}")
elseif(qa_homepage)
set(download_location "${qa_homepage}")
endif()
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
endif()
if(arg_TYPE STREQUAL "SYSTEM_LIBRARY")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
# Try to get package url that was set using CMake's set_package_properties function.
# Relies on querying the internal global property name that CMake sets in its
# implementation.
get_cmake_property(target_url _CMAKE_${package_name_for_spdx_id}_URL)
if(target_url)
set(download_location "${target_url}")
endif()
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(NOT download_location
AND arg_USE_ATTRIBUTION_FILES
AND qa_download_location)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(download_location "${qa_download_location}")
endif()
endif()
if(download_location)
list(APPEND project_package_options DOWNLOAD_LOCATION "${download_location}")
endif()
_qt_internal_sbom_get_package_purpose("${arg_TYPE}" package_purpose)
list(APPEND project_package_options PURPOSE "${package_purpose}")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
set(cpe_values "")
CMake: Add the Qt-specific CPE to 3rd party entity types in SBOM Some vendored 3rd party sources or libraries may not have a CPE because the upstream disappeared, we don't know where the files originally originated from, or no CPE was ever issued. Given that the 3rd party files are shipped with Qt, they can be considered as part of the "Qt" CPE for vulnerability tracking reasons. In such cases, we should add the Qt-specific CPE to the SBOM to ensure that the SBOM is as complete as possible when tooling analyzes the third party packages. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I32a70e24742a860198f3a6b12bdb4a06057f1ab3 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-08-29 18:12:58 +02:00
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(arg_CPE)
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
list(APPEND cpe_values "${arg_CPE}")
CMake: Read CPEs from attribution files when generating SBOMs Also split up the CPE handling to allow for more values to be set, rather than preferring the first one that is encountered. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3209cb5d66f5483c7294b40816431d9df75e00e5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-08-06 18:50:08 +02:00
endif()
CMake: Add the Qt-specific CPE to 3rd party entity types in SBOM Some vendored 3rd party sources or libraries may not have a CPE because the upstream disappeared, we don't know where the files originally originated from, or no CPE was ever issued. Given that the 3rd party files are shipped with Qt, they can be considered as part of the "Qt" CPE for vulnerability tracking reasons. In such cases, we should add the Qt-specific CPE to the SBOM to ensure that the SBOM is as complete as possible when tooling analyzes the third party packages. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I32a70e24742a860198f3a6b12bdb4a06057f1ab3 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-08-29 18:12:58 +02:00
CMake: Read CPEs from attribution files when generating SBOMs Also split up the CPE handling to allow for more values to be set, rather than preferring the first one that is encountered. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3209cb5d66f5483c7294b40816431d9df75e00e5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-08-06 18:50:08 +02:00
if(arg_CPE_VENDOR AND arg_CPE_PRODUCT)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_compute_security_cpe(custom_cpe
VENDOR "${arg_CPE_VENDOR}"
PRODUCT "${arg_CPE_PRODUCT}"
VERSION "${package_version}")
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
list(APPEND cpe_values "${custom_cpe}")
CMake: Read CPEs from attribution files when generating SBOMs Also split up the CPE handling to allow for more values to be set, rather than preferring the first one that is encountered. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3209cb5d66f5483c7294b40816431d9df75e00e5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-08-06 18:50:08 +02:00
endif()
CMake: Add the Qt-specific CPE to 3rd party entity types in SBOM Some vendored 3rd party sources or libraries may not have a CPE because the upstream disappeared, we don't know where the files originally originated from, or no CPE was ever issued. Given that the 3rd party files are shipped with Qt, they can be considered as part of the "Qt" CPE for vulnerability tracking reasons. In such cases, we should add the Qt-specific CPE to the SBOM to ensure that the SBOM is as complete as possible when tooling analyzes the third party packages. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I32a70e24742a860198f3a6b12bdb4a06057f1ab3 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-08-29 18:12:58 +02:00
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg_USE_ATTRIBUTION_FILES AND qa_cpes)
CMake: Avoid warnings _qt_internal_sbom_replace_qa_placeholders We should only pass the VERSION option if we have a value to pass. Fixes warnings like the one below when using CMake 3.31: CMake Warning (dev) at cmake/QtPublicSbomAttributionHelpers.cmake:511 (cmake_parse_arguments): The VERSION keyword was followed by an empty string or no value at all. Policy CMP0174 is not set, so cmake_parse_arguments() will unset the arg_VERSION variable rather than setting it to an empty string. Call Stack (most recent call first): cmake/QtPublicSbomHelpers.cmake:954 (_qt_internal_sbom_replace_qa_placeholders) Amends bc3bbb51b7b48d3c4a44a432441938863582242c Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I671a9153d72b21b6aed2112bea45f577a5cee4eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-02-17 14:59:52 +01:00
set(placeholder_args "")
if(package_version)
list(APPEND placeholder_args VERSION "${package_version}")
endif()
CMake: Replace placeholders in CPE and PURL strings in SBOMs Replace instances of $<VERSION> in CPE and PURL strings read from qt_attribution.json files with the version of the package being processed. This avoids duplicating the version in qt_attribution.json files in 3 different fields Version, CPE, and PURL. Pick-to: 6.8 6.9 Task-number: QTBUG-132181 Change-Id: I91af17c82dbb936739f4811bf86043e00ee49a78 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-03 18:40:23 +01:00
_qt_internal_sbom_replace_qa_placeholders(
VALUES ${qa_cpes}
CMake: Avoid warnings _qt_internal_sbom_replace_qa_placeholders We should only pass the VERSION option if we have a value to pass. Fixes warnings like the one below when using CMake 3.31: CMake Warning (dev) at cmake/QtPublicSbomAttributionHelpers.cmake:511 (cmake_parse_arguments): The VERSION keyword was followed by an empty string or no value at all. Policy CMP0174 is not set, so cmake_parse_arguments() will unset the arg_VERSION variable rather than setting it to an empty string. Call Stack (most recent call first): cmake/QtPublicSbomHelpers.cmake:954 (_qt_internal_sbom_replace_qa_placeholders) Amends bc3bbb51b7b48d3c4a44a432441938863582242c Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I671a9153d72b21b6aed2112bea45f577a5cee4eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-02-17 14:59:52 +01:00
${placeholder_args}
CMake: Replace placeholders in CPE and PURL strings in SBOMs Replace instances of $<VERSION> in CPE and PURL strings read from qt_attribution.json files with the version of the package being processed. This avoids duplicating the version in qt_attribution.json files in 3 different fields Version, CPE, and PURL. Pick-to: 6.8 6.9 Task-number: QTBUG-132181 Change-Id: I91af17c82dbb936739f4811bf86043e00ee49a78 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-03 18:40:23 +01:00
OUT_VAR qa_cpes_replaced
)
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
list(APPEND cpe_values "${qa_cpes_replaced}")
CMake: Read CPEs from attribution files when generating SBOMs Also split up the CPE handling to allow for more values to be set, rather than preferring the first one that is encountered. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I3209cb5d66f5483c7294b40816431d9df75e00e5 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-08-06 18:50:08 +02:00
endif()
CMake: Add the Qt-specific CPE to 3rd party entity types in SBOM Some vendored 3rd party sources or libraries may not have a CPE because the upstream disappeared, we don't know where the files originally originated from, or no CPE was ever issued. Given that the 3rd party files are shipped with Qt, they can be considered as part of the "Qt" CPE for vulnerability tracking reasons. In such cases, we should add the Qt-specific CPE to the SBOM to ensure that the SBOM is as complete as possible when tooling analyzes the third party packages. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I32a70e24742a860198f3a6b12bdb4a06057f1ab3 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-08-29 18:12:58 +02:00
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg___QT_INTERNAL_HANDLE_QT_ENTITY_TYPE_CPE)
_qt_internal_sbom_forward_sbom_add_target_options(sbom_add_target_args)
_qt_internal_sbom_handle_qt_entity_cpe(${target} ${sbom_add_target_args}
CPE "${cpe_values}"
OUT_VAR qt_cpe_list)
if(qt_cpe_list)
list(APPEND cpe_values ${qt_cpe_list})
endif()
CMake: Add the Qt-specific CPE to 3rd party entity types in SBOM Some vendored 3rd party sources or libraries may not have a CPE because the upstream disappeared, we don't know where the files originally originated from, or no CPE was ever issued. Given that the 3rd party files are shipped with Qt, they can be considered as part of the "Qt" CPE for vulnerability tracking reasons. In such cases, we should add the Qt-specific CPE to the SBOM to ensure that the SBOM is as complete as possible when tooling analyzes the third party packages. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I32a70e24742a860198f3a6b12bdb4a06057f1ab3 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-08-29 18:12:58 +02:00
endif()
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(cpe_values)
list(APPEND project_package_options CPE ${cpe_values})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
# Assemble arguments to forward to the function that handles purl options.
set(purl_args "")
_qt_internal_get_sbom_purl_add_target_options(purl_opt_args purl_single_args purl_multi_args)
_qt_internal_forward_function_args(
FORWARD_APPEND
FORWARD_PREFIX arg
FORWARD_OUT_VAR purl_args
FORWARD_OPTIONS
${purl_opt_args}
FORWARD_SINGLE
${purl_single_args}
TYPE
FORWARD_MULTI
${purl_multi_args}
)
CMake: Improve SBOM PURL handling Tools that consume SBOMs expect that each SBOM package will contain a unique purl: https://github.com/package-url/purl-spec Each Qt target maps to an SBOM package, so generate a target-specific PURL for each Qt target, by using the combination of the repo name and target name as the purl name. The purl external reference will then look something like: ExternalRef: PACKAGE-MANAGER purl pkg:/TheQtCompany/qtbase-Gui@6.8.0 Also allow customizing the purl for each target by specifying one of the following options to functions like qt_internal_add_module or qt_internal_extend_sbom: - PURL_TYPE - PURL_NAMESPACE - PURL_NAME - PURL_VERSION - PURL_SUBPATH - PURL_QUALIFIERS - NO_PURL - NO_DEFAULT_QT_PURL Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6926dd773a0ef6fc688664bcac7b23483ecbabe6 Reviewed-by: Kai Köhne <kai.koehne@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-22 17:26:29 +02:00
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
if(supplier)
list(APPEND purl_args SUPPLIER "${supplier}")
endif()
if(package_version)
CMake: Rework SBOM PURL handling Previously, only 3 purl entries could be added to a target, which tightly coupled to Qt's needs: a QT one pointing to code.qt.io, a MIRROR one pointing to github, and an upstream one pointing to some upstream third party location. Rework the implementation to allow for an unlimited number of PURL entries and to allow more flexibility when adding PURLs in a user project. The new syntax for adding PURLs to a target, which is also the basis for a future public API is as follows: qt_internal_add_sbom(<target> PURLS [[PURL_ENTRY PURL_ID <id> PURL_TYPE <type> PURL_NAMESPACE <namespace> PURL_NAME <name> PURL_VERSION <version>]...] PURL_VALUES [purl-string...] ) The PURLS keyword is used to specify multiple PURL entries, each starting with the PURL_ENTRY keyword. The PURL_VALUES keyword is used to specify a list of pre-built purl strings. PURL_ID is an optional argument used to identify a specific purl entry, which is mostly needed for Qt's needs, to post-process them further. The rest of the options are pre-existing from the previous implementation. Implementation-wise, there's a new custom parser to be able to parse and validate PURL_ENTRY arguments. The VERSION option was renamed to PACKAGE_VERSION, to avoid some issues in cmake_parse_arguments parsing with nested VERSION options. The NO_PURL option was removed because it makes no sense in the new implementation, because if you specify some PURL arguments, there is already an intention to generate a PURL entry. Qt entities no longer have a restriction on which specific purl ids they can have. The new Qt specific purl IDs have been renamed: - QT -> GENERIC - MIRROR -> GITHUB Amends f7e1123620b623be0c321b54eaba7a1d618a7ce1 Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I050decece1c6d9e6e0e06547043f864d6f497ea7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-24 18:13:54 +01:00
list(APPEND purl_args PACKAGE_VERSION "${package_version}")
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
endif()
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
CMake: Decouple Qt specific handling from SBOM implementation The SBOM functions so far had a lot of Qt-specific logic inside them. Decouple this logic into separate Qt-specific functions or explicitly guard the code with Qt-specific handling options, to prepare for a cleaner SBOM public API. The generic functions then call the Qt-specific ones if various internal options are set. This approach is used, rather than directly passing values to the generic functions because: - we have cases where we need to recursively pass the values all the way down to all recursively created attribution targets - some of the logic needs to know about values before and after qt processing, and this could be achieved with something like lambdas but it's not worth the complexity Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I4399c41f4d976f20b16a0bb0c674d4f07ee1ccd4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-13 15:30:15 +01:00
if(arg_USE_ATTRIBUTION_FILES AND qa_purls)
CMake: Avoid warnings _qt_internal_sbom_replace_qa_placeholders We should only pass the VERSION option if we have a value to pass. Fixes warnings like the one below when using CMake 3.31: CMake Warning (dev) at cmake/QtPublicSbomAttributionHelpers.cmake:511 (cmake_parse_arguments): The VERSION keyword was followed by an empty string or no value at all. Policy CMP0174 is not set, so cmake_parse_arguments() will unset the arg_VERSION variable rather than setting it to an empty string. Call Stack (most recent call first): cmake/QtPublicSbomHelpers.cmake:954 (_qt_internal_sbom_replace_qa_placeholders) Amends bc3bbb51b7b48d3c4a44a432441938863582242c Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I671a9153d72b21b6aed2112bea45f577a5cee4eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-02-17 14:59:52 +01:00
set(placeholder_args "")
if(package_version)
list(APPEND placeholder_args VERSION "${package_version}")
endif()
CMake: Replace placeholders in CPE and PURL strings in SBOMs Replace instances of $<VERSION> in CPE and PURL strings read from qt_attribution.json files with the version of the package being processed. This avoids duplicating the version in qt_attribution.json files in 3 different fields Version, CPE, and PURL. Pick-to: 6.8 6.9 Task-number: QTBUG-132181 Change-Id: I91af17c82dbb936739f4811bf86043e00ee49a78 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-03 18:40:23 +01:00
_qt_internal_sbom_replace_qa_placeholders(
VALUES ${qa_purls}
CMake: Avoid warnings _qt_internal_sbom_replace_qa_placeholders We should only pass the VERSION option if we have a value to pass. Fixes warnings like the one below when using CMake 3.31: CMake Warning (dev) at cmake/QtPublicSbomAttributionHelpers.cmake:511 (cmake_parse_arguments): The VERSION keyword was followed by an empty string or no value at all. Policy CMP0174 is not set, so cmake_parse_arguments() will unset the arg_VERSION variable rather than setting it to an empty string. Call Stack (most recent call first): cmake/QtPublicSbomHelpers.cmake:954 (_qt_internal_sbom_replace_qa_placeholders) Amends bc3bbb51b7b48d3c4a44a432441938863582242c Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I671a9153d72b21b6aed2112bea45f577a5cee4eb Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-02-17 14:59:52 +01:00
${placeholder_args}
CMake: Replace placeholders in CPE and PURL strings in SBOMs Replace instances of $<VERSION> in CPE and PURL strings read from qt_attribution.json files with the version of the package being processed. This avoids duplicating the version in qt_attribution.json files in 3 different fields Version, CPE, and PURL. Pick-to: 6.8 6.9 Task-number: QTBUG-132181 Change-Id: I91af17c82dbb936739f4811bf86043e00ee49a78 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-03 18:40:23 +01:00
OUT_VAR qa_purls_replaced
)
CMake: Rework SBOM PURL handling Previously, only 3 purl entries could be added to a target, which tightly coupled to Qt's needs: a QT one pointing to code.qt.io, a MIRROR one pointing to github, and an upstream one pointing to some upstream third party location. Rework the implementation to allow for an unlimited number of PURL entries and to allow more flexibility when adding PURLs in a user project. The new syntax for adding PURLs to a target, which is also the basis for a future public API is as follows: qt_internal_add_sbom(<target> PURLS [[PURL_ENTRY PURL_ID <id> PURL_TYPE <type> PURL_NAMESPACE <namespace> PURL_NAME <name> PURL_VERSION <version>]...] PURL_VALUES [purl-string...] ) The PURLS keyword is used to specify multiple PURL entries, each starting with the PURL_ENTRY keyword. The PURL_VALUES keyword is used to specify a list of pre-built purl strings. PURL_ID is an optional argument used to identify a specific purl entry, which is mostly needed for Qt's needs, to post-process them further. The rest of the options are pre-existing from the previous implementation. Implementation-wise, there's a new custom parser to be able to parse and validate PURL_ENTRY arguments. The VERSION option was renamed to PACKAGE_VERSION, to avoid some issues in cmake_parse_arguments parsing with nested VERSION options. The NO_PURL option was removed because it makes no sense in the new implementation, because if you specify some PURL arguments, there is already an intention to generate a PURL entry. Qt entities no longer have a restriction on which specific purl ids they can have. The new Qt specific purl IDs have been renamed: - QT -> GENERIC - MIRROR -> GITHUB Amends f7e1123620b623be0c321b54eaba7a1d618a7ce1 Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I050decece1c6d9e6e0e06547043f864d6f497ea7 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-24 18:13:54 +01:00
list(APPEND purl_args PURL_VALUES ${qa_purls_replaced})
CMake: Read UpstreamPURL from attribution files when generating SBOMs Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id65770cdee17c6bf4701b10565ab428f3e28887f Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-08-06 18:50:08 +02:00
endif()
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
list(APPEND purl_args OUT_VAR purl_package_options)
CMake: Improve SBOM PURL handling Tools that consume SBOMs expect that each SBOM package will contain a unique purl: https://github.com/package-url/purl-spec Each Qt target maps to an SBOM package, so generate a target-specific PURL for each Qt target, by using the combination of the repo name and target name as the purl name. The purl external reference will then look something like: ExternalRef: PACKAGE-MANAGER purl pkg:/TheQtCompany/qtbase-Gui@6.8.0 Also allow customizing the purl for each target by specifying one of the following options to functions like qt_internal_add_module or qt_internal_extend_sbom: - PURL_TYPE - PURL_NAMESPACE - PURL_NAME - PURL_VERSION - PURL_SUBPATH - PURL_QUALIFIERS - NO_PURL - NO_DEFAULT_QT_PURL Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6926dd773a0ef6fc688664bcac7b23483ecbabe6 Reviewed-by: Kai Köhne <kai.koehne@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-22 17:26:29 +02:00
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
_qt_internal_sbom_handle_purl_values(${target} ${purl_args})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
if(purl_package_options)
list(APPEND project_package_options ${purl_package_options})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Remove type restriction on attribution file extra SBOM info So it can be used in other non-Qt projects. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I11d81d2c6819906fffbffe81f4db1b7af28cb99b Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-02-24 15:03:32 +01:00
if(arg_USE_ATTRIBUTION_FILES)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
if(qa_chosen_attribution_file_path)
CMake: Improve handling of file paths embedded into SBOM Filter out source files that are not actually sources, like pkg config files and json.gen files. To make the output more reproducible, replace the absolute source and build directories with relative paths prepended with a marker like /src_dir/ and /build_dir/. Apply this to source file paths and to qt attribution json file paths. Genex wrapped sources still need to be handled, but that will come in a later change. Amends 37a5e001277db9e1392a242171ab2b88cb6c3049 Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-130557 Change-Id: Ic38cd9df827c1da770c1f337bc4725748bed5560 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-25 16:29:59 +02:00
_qt_internal_sbom_map_path_to_reproducible_relative_path(relative_attribution_path
PATH "${qa_chosen_attribution_file_path}"
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
string(APPEND package_comment
CMake: Improve handling of file paths embedded into SBOM Filter out source files that are not actually sources, like pkg config files and json.gen files. To make the output more reproducible, replace the absolute source and build directories with relative paths prepended with a marker like /src_dir/ and /build_dir/. Apply this to source file paths and to qt attribution json file paths. Genex wrapped sources still need to be handled, but that will come in a later change. Amends 37a5e001277db9e1392a242171ab2b88cb6c3049 Pick-to: 6.8 Task-number: QTBUG-122899 Task-number: QTBUG-130557 Change-Id: Ic38cd9df827c1da770c1f337bc4725748bed5560 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-25 16:29:59 +02:00
" Information extracted from:\n ${relative_attribution_path}\n")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
if(NOT "${qa_chosen_attribution_entry_index}" STREQUAL "")
string(APPEND package_comment
" Entry index: ${qa_chosen_attribution_entry_index}\n")
endif()
CMake: Re-order SBOM embedded qt_attribution.json fields Put the attribution file path and entry first, followed by the id, and the rest of the fields. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I478a48980e6207646189d6b80ec57dbc9efb4902 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-02-24 18:07:22 +01:00
if(qa_attribution_id)
string(APPEND package_comment " Id: ${qa_attribution_id}\n")
endif()
if(qa_attribution_name)
string(APPEND package_comment " Name: ${qa_attribution_name}\n")
endif()
if(qa_description)
string(APPEND package_comment " Description: ${qa_description}\n")
endif()
if(qa_qt_usage)
string(APPEND package_comment " Qt usage: ${qa_qt_usage}\n")
endif()
CMake: Include license and license file attribution in SBOM comment Some attribution entries don't have a SPDX license id specified, in that case it's good to at least include the free-form license name and file path. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I75bb5c30645684ea74fe94da92ea30eb29965ad4 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-02-27 15:43:08 +01:00
if(qa_license)
string(APPEND package_comment " License: ${qa_license}\n")
endif()
if(qa_license_file)
string(APPEND package_comment " License file: ${qa_license_file}\n")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
if(package_comment)
list(APPEND project_package_options COMMENT "<text>\n${package_comment}</text>")
endif()
_qt_internal_sbom_handle_target_dependencies("${target}"
SPDX_ID "${package_spdx_id}"
LIBRARIES "${arg_LIBRARIES}"
PUBLIC_LIBRARIES "${arg_PUBLIC_LIBRARIES}"
OUT_RELATIONSHIPS relationships
)
get_cmake_property(project_spdx_id _qt_internal_sbom_project_spdx_id)
list(APPEND relationships "${project_spdx_id} CONTAINS ${package_spdx_id}")
CMake: Allow adding custom SBOM relationships to targets and projects Add a new SBOM_RELATIONSHIPS option to qt_internal_extend_target and friends that allows adding custom relationships to the current target. Add a new function qt_internal_sbom_add_project_relationship that allows adding custom relationships to the current project SBOM document. A sample usage might be: qt_internal_sbom_get_project_spdx_id(project_spdx_id) qt_internal_sbom_get_target_spdx_id(Svg svg_spdx_id) qt_internal_extend_target(Svg SBOM_RELATIONSHIPS "${svg_spdx_id} DESCENDANT_OF ${project_spdx_id}" ) qt_internal_sbom_add_project_relationship( RELATIONSHIPS "${svg_spdx_id} CONTAINS NOASSERTION" "${svg_spdx_id} DESCRIBES NOASSERTION" "${project_spdx_id} DESCRIBES NOASSERTION" ) Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-129901 Task-number: QTBUG-131377 Change-Id: Ie0119ca71b047c7515e1deaf84a5a67ea01b5274 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-12-02 19:13:29 +01:00
if(arg_SBOM_RELATIONSHIPS)
list(APPEND relationships "${arg_SBOM_RELATIONSHIPS}")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
list(REMOVE_DUPLICATES relationships)
list(JOIN relationships "\nRelationship: " relationships)
list(APPEND project_package_options RELATIONSHIP "${relationships}")
_qt_internal_sbom_generate_add_package(
PACKAGE "${package_name_for_spdx_id}"
SPDXID "${package_spdx_id}"
CONTAINS_FILES
${project_package_options}
)
set(no_install_option "")
if(arg_NO_INSTALL)
set(no_install_option NO_INSTALL)
endif()
set(framework_option "")
if(APPLE AND NOT target_type STREQUAL "INTERFACE_LIBRARY")
get_target_property(is_framework ${target} FRAMEWORK)
if(is_framework)
set(framework_option "FRAMEWORK")
endif()
endif()
set(install_prefix_option "")
get_cmake_property(install_prefix _qt_internal_sbom_install_prefix)
if(install_prefix)
set(install_prefix_option INSTALL_PREFIX "${install_prefix}")
endif()
_qt_internal_forward_function_args(
FORWARD_PREFIX arg
FORWARD_OUT_VAR target_binary_multi_config_args
FORWARD_SINGLE
${multi_config_single_args}
)
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
set(copyrights_option "")
if(copyrights)
set(copyrights_option COPYRIGHTS "${copyrights}")
endif()
set(license_option "")
if(license_expression)
set(license_option LICENSE_EXPRESSION "${license_expression}")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_sbom_handle_target_binary_files("${target}"
${no_install_option}
${framework_option}
${install_prefix_option}
TYPE "${arg_TYPE}"
${target_binary_multi_config_args}
SPDX_ID "${package_spdx_id}"
CMake: Fix CMP0174 warnings in SBOM generation code Pick-to: 6.8 Change-Id: Iabe959eda65a0bf7dd94b20cfce7d55642f9096d Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-11-19 15:47:14 +01:00
${copyrights_option}
${license_option}
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
CMake: Add SBOM support for translations, custom files and friends This change introduces a new SBOM API to add info about various kinds of files to a target SBOM package. The new function name is called qt_internal_sbom_add_files(). The motivating case is including SBOM information for qt translation files, resources (e.g. webengine data files) and any other kind of custom files. A sample call might look like: qt_internal_sbom_add_files(Translations FILES "${qm_files}" SOURCE_FILES_ONE_PER_INPUT_FILE "${ts_files}" FILE_TYPE "QT_TRANSLATION" INSTALL_PATH "${INSTALL_TRANSLATIONSDIR}" ) While the motivating case is Qt-specific, the function implementation is being somewhat future proofed for the not-yet created public SBOM API. The new API supports adding files to any target that is backed by an SBOM package, so all targets created by qt_internal_add_module() and friends, as well as ones created by qt_internal_add_sbom(). It can be called multiple times for the same target, with a different set of files, to e.g. assign a different license, or file type per file set. Note that the file set doesn't have anything to do with CMake's concept of file sets. The function is also multi-config aware, and allows specifying different install paths per config, as well as generator expressions in file names. But the multi-config support is a bit wonky, and might need some rethinking in the future. Note that the custom files must be installed and available in the specified qt install path, because the file contents will be checksummed at install time and embedded into the sbom document. Calling the new API does not do installation itself. Implementation wise, the function call flow is - project calls qt_internal_sbom_add_files() one or more times - at finalization time, the _qt_internal_sbom_add_target finalizer is called for a target, which then calls _qt_internal_sbom_handle_target_custom_files() - the latter calls _qt_internal_sbom_handle_target_custom_file_set() for each file set that was added to the target - the latter calls _qt_internal_sbom_handle_multi_config_custom_file() for each input file in the file set, which ultimately calls _qt_internal_sbom_add_custom_file() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-128320 Change-Id: Iafde26ebd68f4168b49e55fbc8ad1c251e98d4b0 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-11-18 17:12:17 +01:00
_qt_internal_sbom_handle_target_custom_files("${target}"
${no_install_option}
${install_prefix_option}
PACKAGE_TYPE "${arg_TYPE}"
PACKAGE_SPDX_ID "${package_spdx_id}"
${copyrights_option}
${license_option}
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endfunction()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Helper to add sbom information for a possibly non-existing target.
# This will defer the actual sbom generation until the end of the directory scope, unless
# immediate finalization was requested.
function(_qt_internal_add_sbom target)
if(NOT QT_GENERATE_SBOM)
return()
endif()
set(opt_args
IMMEDIATE_FINALIZATION
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(single_args
TYPE
FRIENDLY_PACKAGE_NAME
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(multi_args "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# No validation on purpose, the other options will be validated later.
set(forward_args ${ARGN})
# Remove the IMMEDIATE_FINALIZATION from the forwarded args.
list(REMOVE_ITEM forward_args IMMEDIATE_FINALIZATION)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# If a target doesn't exist we create it.
if(NOT TARGET "${target}")
_qt_internal_create_sbom_target("${target}" ${forward_args})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Save the passed options.
_qt_internal_extend_sbom("${target}" ${forward_args})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Defer finalization. In case it was already deferred, it will be a no-op.
# Some targets need immediate finalization, like the PlatformInternal ones, because otherwise
# they would be finalized after the sbom was already generated.
set(immediate_finalization "")
if(arg_IMMEDIATE_FINALIZATION)
set(immediate_finalization IMMEDIATE_FINALIZATION)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
_qt_internal_defer_sbom_finalization("${target}" ${immediate_finalization})
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Helper to add custom sbom information for some kind of dependency that is not backed by an
# existing target.
# Useful for cases like 3rd party dependencies not represented by an already existing imported
# target, or for 3rd party sources that get compiled into a regular Qt target (PCRE sources compiled
# into Bootstrap).
function(_qt_internal_create_sbom_target target)
if(NOT QT_GENERATE_SBOM)
return()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(opt_args "")
set(single_args
TYPE
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
# No validation on purpose, the other options will be validated later.
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(TARGET "${target}")
message(FATAL_ERROR "The target ${target} already exists.")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
add_library("${target}" INTERFACE IMPORTED)
set_target_properties(${target} PROPERTIES
_qt_sbom_is_custom_sbom_target "TRUE"
IMPORTED_GLOBAL TRUE
)
if(NOT arg_TYPE)
message(FATAL_ERROR "No SBOM TYPE option was provided for target: ${target}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Helper to add additional sbom information for an existing target.
# Just appends the options to the target's sbom args property, which will will be evaluated
# during finalization.
function(_qt_internal_extend_sbom target)
if(NOT QT_GENERATE_SBOM)
return()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(NOT TARGET "${target}")
message(FATAL_ERROR
"The target ${target} does not exist, use qt_internal_add_sbom to create "
"a target first, or call the function on any other exsiting target.")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(opt_args "")
set(single_args
TYPE
FRIENDLY_PACKAGE_NAME
)
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
# No validation on purpose, the other options will be validated later.
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Make sure a spdx id is recorded for the target right now, so it is "known" when handling
# relationships for other targets, even if the target was not yet finalized.
if(arg_TYPE)
# Friendly package name is allowed to be empty.
set(package_name_option "")
if(arg_FRIENDLY_PACKAGE_NAME)
set(package_name_option PACKAGE_NAME "${arg_FRIENDLY_PACKAGE_NAME}")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
_qt_internal_sbom_record_target_spdx_id(${target}
TYPE "${arg_TYPE}"
${package_name_option}
)
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set_property(TARGET ${target} APPEND PROPERTY _qt_finalize_sbom_args "${ARGN}")
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Helper to add additional sbom information to targets created by qt_find_package.
# If the package was not found, and the targets were not created, the functions does nothing.
# This is similar to _qt_internal_extend_sbom, but is explicit in the fact that the targets might
# not exist.
function(_qt_find_package_extend_sbom)
if(NOT QT_GENERATE_SBOM)
return()
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
_qt_internal_get_sbom_add_target_options(sbom_opt_args sbom_single_args sbom_multi_args)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(opt_args
${sbom_opt_args}
)
set(single_args
${sbom_single_args}
)
set(multi_args
TARGETS
${sbom_multi_args}
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
cmake_parse_arguments(PARSE_ARGV 0 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Make sure not to forward TARGETS.
set(sbom_args "")
_qt_internal_forward_function_args(
FORWARD_APPEND
FORWARD_PREFIX arg
FORWARD_OUT_VAR sbom_args
FORWARD_OPTIONS
${sbom_opt_args}
FORWARD_SINGLE
${sbom_single_args}
FORWARD_MULTI
${sbom_multi_args}
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
foreach(target IN LISTS arg_TARGETS)
if(TARGET "${target}")
_qt_internal_extend_sbom("${target}" ${sbom_args})
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
else()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
message(DEBUG "The target ${target} does not exist, skipping extending the sbom info.")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
endforeach()
endfunction()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Helper to defer adding sbom information for a target, at the end of the directory scope.
function(_qt_internal_defer_sbom_finalization target)
if(NOT QT_GENERATE_SBOM)
return()
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(opt_args
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
IMMEDIATE_FINALIZATION
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(single_args "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
get_target_property(sbom_finalization_requested ${target} _qt_sbom_finalization_requested)
if(sbom_finalization_requested)
# Already requested, nothing to do.
return()
endif()
set_target_properties(${target} PROPERTIES _qt_sbom_finalization_requested TRUE)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
_qt_internal_append_to_cmake_property_without_duplicates(
_qt_internal_sbom_targets_waiting_for_finalization
"${target}"
)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(func "_qt_internal_finalize_sbom")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(arg_IMMEDIATE_FINALIZATION)
_qt_internal_finalize_sbom(${target})
elseif(QT_BUILDING_QT)
qt_add_list_file_finalizer("${func}" "${target}")
elseif(CMAKE_VERSION VERSION_GREATER_EQUAL "3.19")
cmake_language(EVAL CODE "cmake_language(DEFER CALL \"${func}\" \"${target}\")")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
else()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
message(FATAL_ERROR "Defer adding a sbom target requires CMake version 3.19")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
endfunction()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Finalizer to add sbom information for the target.
# Expects the target to exist.
function(_qt_internal_finalize_sbom target)
if(NOT QT_GENERATE_SBOM)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
return()
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
get_target_property(sbom_finalization_done ${target} _qt_sbom_finalization_done)
if(sbom_finalization_done)
# Already done, nothing to do.
return()
endif()
set_target_properties(${target} PROPERTIES _qt_sbom_finalization_done TRUE)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
get_target_property(sbom_args ${target} _qt_finalize_sbom_args)
if(NOT sbom_args)
set(sbom_args "")
endif()
_qt_internal_sbom_add_target(${target} ${sbom_args})
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Extends the list of targets that are considered dependencies for target.
function(_qt_internal_extend_sbom_dependencies target)
if(NOT QT_GENERATE_SBOM)
return()
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(opt_args "")
set(single_args "")
set(multi_args
SBOM_DEPENDENCIES
)
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(NOT TARGET "${target}")
message(FATAL_ERROR "The target ${target} does not exist.")
endif()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
_qt_internal_append_to_target_property_without_duplicates(${target}
_qt_extend_target_sbom_dependencies "${arg_SBOM_DEPENDENCIES}"
)
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Sets the sbom project name for the root project.
function(_qt_internal_sbom_set_root_project_name project_name)
set_property(GLOBAL PROPERTY _qt_internal_sbom_repo_project_name "${project_name}")
endfunction()
CMake: Make ninja 'sbom' work for top-level builds We need to pass all dirs where sboms will be generated, for external document referencing. We also need to set up dependencies between the repo sbom custom targets, so that qtsvg sbom is only run after qtbase sbom. We use the dependencies.yaml info to set up the dependencies. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id3331e11742bc2c86e7ed52ce26b3ff21eace359 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-20 19:21:53 +02:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Sets the real qt repo project name for a given project (e.g. set QtWebEngine for project QtPdf).
# This is needed to be able to extract the qt repo dependencies in a top-level build.
function(_qt_internal_sbom_set_qt_repo_project_name project_name)
set_property(GLOBAL PROPERTY _qt_internal_sbom_qt_repo_project_name "${project_name}")
endfunction()
CMake: Make ninja 'sbom' work for top-level builds We need to pass all dirs where sboms will be generated, for external document referencing. We also need to set up dependencies between the repo sbom custom targets, so that qtsvg sbom is only run after qtbase sbom. We use the dependencies.yaml info to set up the dependencies. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Id3331e11742bc2c86e7ed52ce26b3ff21eace359 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-06-20 19:21:53 +02:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Get repo project_name spdx id reference, needs to start with Package- to be NTIA compliant.
function(_qt_internal_sbom_get_root_project_name_for_spdx_id out_var)
_qt_internal_sbom_get_root_project_name_lower_case(repo_project_name_lowercase)
set(sbom_repo_project_name "Package-${repo_project_name_lowercase}")
set(${out_var} "${sbom_repo_project_name}" PARENT_SCOPE)
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Returns the lower case sbom project name.
function(_qt_internal_sbom_get_root_project_name_lower_case out_var)
get_cmake_property(project_name _qt_internal_sbom_repo_project_name)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(NOT project_name)
message(FATAL_ERROR "No SBOM project name was set.")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
string(TOLOWER "${project_name}" repo_project_name_lowercase)
set(${out_var} "${repo_project_name_lowercase}" PARENT_SCOPE)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endfunction()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Returns the lower case real qt repo project name (e.g. returns 'qtwebengine' when building the
# project qtpdf).
function(_qt_internal_sbom_get_qt_repo_project_name_lower_case out_var)
get_cmake_property(project_name _qt_internal_sbom_qt_repo_project_name)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(NOT project_name)
message(FATAL_ERROR "No real Qt repo SBOM project name was set.")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
string(TOLOWER "${project_name}" repo_project_name_lowercase)
set(${out_var} "${repo_project_name_lowercase}" PARENT_SCOPE)
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Get a spdx id to reference an external document.
function(_qt_internal_sbom_get_external_document_ref_spdx_id repo_name out_var)
set(${out_var} "DocumentRef-${repo_name}" PARENT_SCOPE)
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Sanitize a given value to be used as a SPDX id.
function(_qt_internal_sbom_get_sanitized_spdx_id out_var hint)
# Only allow alphanumeric characters and dashes.
string(REGEX REPLACE "[^a-zA-Z0-9]+" "-" spdx_id "${hint}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Remove all trailing dashes.
string(REGEX REPLACE "-+$" "" spdx_id "${spdx_id}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(${out_var} "${spdx_id}" PARENT_SCOPE)
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Generates a spdx id for a target and saves it its properties.
function(_qt_internal_sbom_record_target_spdx_id target)
set(opt_args "")
set(single_args
PACKAGE_NAME
TYPE
OUT_VAR
)
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
_qt_internal_sbom_get_spdx_id_for_target("${target}" spdx_id)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(spdx_id)
# Return early if the target was already recorded and has a spdx id.
if(arg_OUT_VAR)
set(${arg_OUT_VAR} "${spdx_id}" PARENT_SCOPE)
endif()
CMake: Skip handling binary files in SBOM for excluded targets If a target has the _qt_internal_excluded_from_default_target property set, don't try to add file SBOM information for the target, because the file will not be built nor installed by default. We check for a new custom _qt_internal_excluded_from_default_target property instead of EXCLUDE_FROM_ALL, because EXCLUDE_FROM_ALL might be set to a genex that excludes all configs except the main one, but we are interested whether the target is entirely excluded. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I79d6a4b0c65356da14f7ff50ee3639705f5fabbd Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-06-21 16:25:05 +02:00
return()
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(arg_PACKAGE_NAME)
set(package_name_for_spdx_id "${arg_PACKAGE_NAME}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
else()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(package_name_for_spdx_id "${target}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
_qt_internal_sbom_generate_target_package_spdx_id(package_spdx_id
TYPE "${arg_TYPE}"
PACKAGE_NAME "${package_name_for_spdx_id}"
)
_qt_internal_sbom_save_spdx_id_for_target("${target}" "${package_spdx_id}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
_qt_internal_sbom_is_qt_entity_type("${arg_TYPE}" is_qt_entity_type)
_qt_internal_sbom_save_spdx_id_for_qt_entity_type(
"${target}" "${is_qt_entity_type}" "${package_spdx_id}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(arg_OUT_VAR)
set(${arg_OUT_VAR} "${package_spdx_id}" PARENT_SCOPE)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
endfunction()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Generates a sanitized spdx id for a target (package) of a specific type.
function(_qt_internal_sbom_generate_target_package_spdx_id out_var)
set(opt_args "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
set(single_args
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
PACKAGE_NAME
TYPE
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
_qt_internal_validate_all_args_are_parsed(arg)
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(NOT arg_PACKAGE_NAME)
message(FATAL_ERROR "PACKAGE_NAME must be set")
endif()
if(NOT arg_TYPE)
message(FATAL_ERROR "TYPE must be set")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
_qt_internal_sbom_get_root_project_name_for_spdx_id(repo_project_name_spdx_id)
_qt_internal_sbom_get_package_infix("${arg_TYPE}" package_infix)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
_qt_internal_sbom_get_sanitized_spdx_id(spdx_id
"SPDXRef-${repo_project_name_spdx_id}-${package_infix}-${arg_PACKAGE_NAME}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(${out_var} "${spdx_id}" PARENT_SCOPE)
endfunction()
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Save a spdx id for a target inside its target properties.
# Also saves the repo document namespace and relative installed repo document path.
# These are used when generating a SPDX external document reference for exported targets, to
# include them in relationships.
function(_qt_internal_sbom_save_spdx_id_for_target target spdx_id)
message(DEBUG "Saving spdx id for target ${target}: ${spdx_id}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(target_unaliased "${target}")
Use `_qt_internal_dealias_target` Pick-to: 6.9 6.8 Change-Id: Iffc1148ed7655c724ea0609be01978ab7d5e7f8f Reviewed-by: Alexandru Croitor <alexandru.croitor@qt.io>
2025-03-05 12:17:44 +01:00
_qt_internal_dealias_target(target_unaliased)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set_target_properties(${target_unaliased} PROPERTIES
_qt_sbom_spdx_id "${spdx_id}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Retrieve repo specific properties.
get_property(repo_document_namespace
GLOBAL PROPERTY _qt_internal_sbom_repo_document_namespace)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
get_property(relative_installed_repo_document_path
GLOBAL PROPERTY _qt_internal_sbom_relative_installed_repo_document_path)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
get_property(project_name_lowercase
GLOBAL PROPERTY _qt_internal_sbom_repo_project_name_lowercase)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# And save them on the target.
set_property(TARGET ${target_unaliased} PROPERTY
_qt_sbom_spdx_repo_document_namespace
"${repo_document_namespace}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set_property(TARGET ${target_unaliased} PROPERTY
_qt_sbom_spdx_relative_installed_repo_document_path
"${relative_installed_repo_document_path}")
set_property(TARGET ${target_unaliased} PROPERTY
_qt_sbom_spdx_repo_project_name_lowercase
"${project_name_lowercase}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Export the properties, so they can be queried by other repos.
# We also do it for versionless targets.
set(export_properties
_qt_sbom_spdx_id
_qt_sbom_spdx_repo_document_namespace
_qt_sbom_spdx_relative_installed_repo_document_path
_qt_sbom_spdx_repo_project_name_lowercase
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
)
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set_property(TARGET "${target_unaliased}" APPEND PROPERTY
EXPORT_PROPERTIES "${export_properties}")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endfunction()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Returns whether the given sbom type is considered to be a Qt type like a module or a tool.
function(_qt_internal_sbom_is_qt_entity_type sbom_type out_var)
set(qt_entity_types
QT_MODULE
QT_PLUGIN
QT_APP
QT_TOOL
QT_TRANSLATIONS
QT_RESOURCES
QT_CUSTOM
QT_CUSTOM_NO_INFIX
)
CMake: Add SBOM support for translations, custom files and friends This change introduces a new SBOM API to add info about various kinds of files to a target SBOM package. The new function name is called qt_internal_sbom_add_files(). The motivating case is including SBOM information for qt translation files, resources (e.g. webengine data files) and any other kind of custom files. A sample call might look like: qt_internal_sbom_add_files(Translations FILES "${qm_files}" SOURCE_FILES_ONE_PER_INPUT_FILE "${ts_files}" FILE_TYPE "QT_TRANSLATION" INSTALL_PATH "${INSTALL_TRANSLATIONSDIR}" ) While the motivating case is Qt-specific, the function implementation is being somewhat future proofed for the not-yet created public SBOM API. The new API supports adding files to any target that is backed by an SBOM package, so all targets created by qt_internal_add_module() and friends, as well as ones created by qt_internal_add_sbom(). It can be called multiple times for the same target, with a different set of files, to e.g. assign a different license, or file type per file set. Note that the file set doesn't have anything to do with CMake's concept of file sets. The function is also multi-config aware, and allows specifying different install paths per config, as well as generator expressions in file names. But the multi-config support is a bit wonky, and might need some rethinking in the future. Note that the custom files must be installed and available in the specified qt install path, because the file contents will be checksummed at install time and embedded into the sbom document. Calling the new API does not do installation itself. Implementation wise, the function call flow is - project calls qt_internal_sbom_add_files() one or more times - at finalization time, the _qt_internal_sbom_add_target finalizer is called for a target, which then calls _qt_internal_sbom_handle_target_custom_files() - the latter calls _qt_internal_sbom_handle_target_custom_file_set() for each file set that was added to the target - the latter calls _qt_internal_sbom_handle_multi_config_custom_file() for each input file in the file set, which ultimately calls _qt_internal_sbom_add_custom_file() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-128320 Change-Id: Iafde26ebd68f4168b49e55fbc8ad1c251e98d4b0 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-11-18 17:12:17 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(is_qt_entity_type FALSE)
if(sbom_type IN_LIST qt_entity_types)
set(is_qt_entity_type TRUE)
CMake: Add SBOM support for translations, custom files and friends This change introduces a new SBOM API to add info about various kinds of files to a target SBOM package. The new function name is called qt_internal_sbom_add_files(). The motivating case is including SBOM information for qt translation files, resources (e.g. webengine data files) and any other kind of custom files. A sample call might look like: qt_internal_sbom_add_files(Translations FILES "${qm_files}" SOURCE_FILES_ONE_PER_INPUT_FILE "${ts_files}" FILE_TYPE "QT_TRANSLATION" INSTALL_PATH "${INSTALL_TRANSLATIONSDIR}" ) While the motivating case is Qt-specific, the function implementation is being somewhat future proofed for the not-yet created public SBOM API. The new API supports adding files to any target that is backed by an SBOM package, so all targets created by qt_internal_add_module() and friends, as well as ones created by qt_internal_add_sbom(). It can be called multiple times for the same target, with a different set of files, to e.g. assign a different license, or file type per file set. Note that the file set doesn't have anything to do with CMake's concept of file sets. The function is also multi-config aware, and allows specifying different install paths per config, as well as generator expressions in file names. But the multi-config support is a bit wonky, and might need some rethinking in the future. Note that the custom files must be installed and available in the specified qt install path, because the file contents will be checksummed at install time and embedded into the sbom document. Calling the new API does not do installation itself. Implementation wise, the function call flow is - project calls qt_internal_sbom_add_files() one or more times - at finalization time, the _qt_internal_sbom_add_target finalizer is called for a target, which then calls _qt_internal_sbom_handle_target_custom_files() - the latter calls _qt_internal_sbom_handle_target_custom_file_set() for each file set that was added to the target - the latter calls _qt_internal_sbom_handle_multi_config_custom_file() for each input file in the file set, which ultimately calls _qt_internal_sbom_add_custom_file() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-128320 Change-Id: Iafde26ebd68f4168b49e55fbc8ad1c251e98d4b0 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-11-18 17:12:17 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(${out_var} ${is_qt_entity_type} PARENT_SCOPE)
CMake: Add SBOM support for translations, custom files and friends This change introduces a new SBOM API to add info about various kinds of files to a target SBOM package. The new function name is called qt_internal_sbom_add_files(). The motivating case is including SBOM information for qt translation files, resources (e.g. webengine data files) and any other kind of custom files. A sample call might look like: qt_internal_sbom_add_files(Translations FILES "${qm_files}" SOURCE_FILES_ONE_PER_INPUT_FILE "${ts_files}" FILE_TYPE "QT_TRANSLATION" INSTALL_PATH "${INSTALL_TRANSLATIONSDIR}" ) While the motivating case is Qt-specific, the function implementation is being somewhat future proofed for the not-yet created public SBOM API. The new API supports adding files to any target that is backed by an SBOM package, so all targets created by qt_internal_add_module() and friends, as well as ones created by qt_internal_add_sbom(). It can be called multiple times for the same target, with a different set of files, to e.g. assign a different license, or file type per file set. Note that the file set doesn't have anything to do with CMake's concept of file sets. The function is also multi-config aware, and allows specifying different install paths per config, as well as generator expressions in file names. But the multi-config support is a bit wonky, and might need some rethinking in the future. Note that the custom files must be installed and available in the specified qt install path, because the file contents will be checksummed at install time and embedded into the sbom document. Calling the new API does not do installation itself. Implementation wise, the function call flow is - project calls qt_internal_sbom_add_files() one or more times - at finalization time, the _qt_internal_sbom_add_target finalizer is called for a target, which then calls _qt_internal_sbom_handle_target_custom_files() - the latter calls _qt_internal_sbom_handle_target_custom_file_set() for each file set that was added to the target - the latter calls _qt_internal_sbom_handle_multi_config_custom_file() for each input file in the file set, which ultimately calls _qt_internal_sbom_add_custom_file() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-128320 Change-Id: Iafde26ebd68f4168b49e55fbc8ad1c251e98d4b0 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-11-18 17:12:17 +01:00
endfunction()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Returns whether the given sbom type is considered to a Qt 3rd party entity type.
function(_qt_internal_sbom_is_qt_3rd_party_entity_type sbom_type out_var)
set(entity_types
QT_THIRD_PARTY_MODULE
QT_THIRD_PARTY_SOURCES
CMake: Add SBOM support for translations, custom files and friends This change introduces a new SBOM API to add info about various kinds of files to a target SBOM package. The new function name is called qt_internal_sbom_add_files(). The motivating case is including SBOM information for qt translation files, resources (e.g. webengine data files) and any other kind of custom files. A sample call might look like: qt_internal_sbom_add_files(Translations FILES "${qm_files}" SOURCE_FILES_ONE_PER_INPUT_FILE "${ts_files}" FILE_TYPE "QT_TRANSLATION" INSTALL_PATH "${INSTALL_TRANSLATIONSDIR}" ) While the motivating case is Qt-specific, the function implementation is being somewhat future proofed for the not-yet created public SBOM API. The new API supports adding files to any target that is backed by an SBOM package, so all targets created by qt_internal_add_module() and friends, as well as ones created by qt_internal_add_sbom(). It can be called multiple times for the same target, with a different set of files, to e.g. assign a different license, or file type per file set. Note that the file set doesn't have anything to do with CMake's concept of file sets. The function is also multi-config aware, and allows specifying different install paths per config, as well as generator expressions in file names. But the multi-config support is a bit wonky, and might need some rethinking in the future. Note that the custom files must be installed and available in the specified qt install path, because the file contents will be checksummed at install time and embedded into the sbom document. Calling the new API does not do installation itself. Implementation wise, the function call flow is - project calls qt_internal_sbom_add_files() one or more times - at finalization time, the _qt_internal_sbom_add_target finalizer is called for a target, which then calls _qt_internal_sbom_handle_target_custom_files() - the latter calls _qt_internal_sbom_handle_target_custom_file_set() for each file set that was added to the target - the latter calls _qt_internal_sbom_handle_multi_config_custom_file() for each input file in the file set, which ultimately calls _qt_internal_sbom_add_custom_file() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-128320 Change-Id: Iafde26ebd68f4168b49e55fbc8ad1c251e98d4b0 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-11-18 17:12:17 +01:00
)
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(is_qt_third_party_entity_type FALSE)
if(sbom_type IN_LIST entity_types)
set(is_qt_third_party_entity_type TRUE)
CMake: Add SBOM support for translations, custom files and friends This change introduces a new SBOM API to add info about various kinds of files to a target SBOM package. The new function name is called qt_internal_sbom_add_files(). The motivating case is including SBOM information for qt translation files, resources (e.g. webengine data files) and any other kind of custom files. A sample call might look like: qt_internal_sbom_add_files(Translations FILES "${qm_files}" SOURCE_FILES_ONE_PER_INPUT_FILE "${ts_files}" FILE_TYPE "QT_TRANSLATION" INSTALL_PATH "${INSTALL_TRANSLATIONSDIR}" ) While the motivating case is Qt-specific, the function implementation is being somewhat future proofed for the not-yet created public SBOM API. The new API supports adding files to any target that is backed by an SBOM package, so all targets created by qt_internal_add_module() and friends, as well as ones created by qt_internal_add_sbom(). It can be called multiple times for the same target, with a different set of files, to e.g. assign a different license, or file type per file set. Note that the file set doesn't have anything to do with CMake's concept of file sets. The function is also multi-config aware, and allows specifying different install paths per config, as well as generator expressions in file names. But the multi-config support is a bit wonky, and might need some rethinking in the future. Note that the custom files must be installed and available in the specified qt install path, because the file contents will be checksummed at install time and embedded into the sbom document. Calling the new API does not do installation itself. Implementation wise, the function call flow is - project calls qt_internal_sbom_add_files() one or more times - at finalization time, the _qt_internal_sbom_add_target finalizer is called for a target, which then calls _qt_internal_sbom_handle_target_custom_files() - the latter calls _qt_internal_sbom_handle_target_custom_file_set() for each file set that was added to the target - the latter calls _qt_internal_sbom_handle_multi_config_custom_file() for each input file in the file set, which ultimately calls _qt_internal_sbom_add_custom_file() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-128320 Change-Id: Iafde26ebd68f4168b49e55fbc8ad1c251e98d4b0 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-11-18 17:12:17 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(${out_var} ${is_qt_third_party_entity_type} PARENT_SCOPE)
endfunction()
CMake: Add SBOM support for translations, custom files and friends This change introduces a new SBOM API to add info about various kinds of files to a target SBOM package. The new function name is called qt_internal_sbom_add_files(). The motivating case is including SBOM information for qt translation files, resources (e.g. webengine data files) and any other kind of custom files. A sample call might look like: qt_internal_sbom_add_files(Translations FILES "${qm_files}" SOURCE_FILES_ONE_PER_INPUT_FILE "${ts_files}" FILE_TYPE "QT_TRANSLATION" INSTALL_PATH "${INSTALL_TRANSLATIONSDIR}" ) While the motivating case is Qt-specific, the function implementation is being somewhat future proofed for the not-yet created public SBOM API. The new API supports adding files to any target that is backed by an SBOM package, so all targets created by qt_internal_add_module() and friends, as well as ones created by qt_internal_add_sbom(). It can be called multiple times for the same target, with a different set of files, to e.g. assign a different license, or file type per file set. Note that the file set doesn't have anything to do with CMake's concept of file sets. The function is also multi-config aware, and allows specifying different install paths per config, as well as generator expressions in file names. But the multi-config support is a bit wonky, and might need some rethinking in the future. Note that the custom files must be installed and available in the specified qt install path, because the file contents will be checksummed at install time and embedded into the sbom document. Calling the new API does not do installation itself. Implementation wise, the function call flow is - project calls qt_internal_sbom_add_files() one or more times - at finalization time, the _qt_internal_sbom_add_target finalizer is called for a target, which then calls _qt_internal_sbom_handle_target_custom_files() - the latter calls _qt_internal_sbom_handle_target_custom_file_set() for each file set that was added to the target - the latter calls _qt_internal_sbom_handle_multi_config_custom_file() for each input file in the file set, which ultimately calls _qt_internal_sbom_add_custom_file() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-128320 Change-Id: Iafde26ebd68f4168b49e55fbc8ad1c251e98d4b0 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-11-18 17:12:17 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Save a spdx id for all known related target names of a given Qt target.
# Related being the namespaced and versionless variants of a Qt target.
# All the related targets will contain the same spdx id.
# So Core, CorePrivate, Qt6::Core, Qt6::CorePrivate, Qt::Core, Qt::CorePrivate will all be
# referred to by the same spdx id.
function(_qt_internal_sbom_save_spdx_id_for_qt_entity_type target is_qt_entity_type package_spdx_id)
# Assign the spdx id to all known related target names of given the given Qt target.
set(target_names "")
CMake: Add SBOM support for translations, custom files and friends This change introduces a new SBOM API to add info about various kinds of files to a target SBOM package. The new function name is called qt_internal_sbom_add_files(). The motivating case is including SBOM information for qt translation files, resources (e.g. webengine data files) and any other kind of custom files. A sample call might look like: qt_internal_sbom_add_files(Translations FILES "${qm_files}" SOURCE_FILES_ONE_PER_INPUT_FILE "${ts_files}" FILE_TYPE "QT_TRANSLATION" INSTALL_PATH "${INSTALL_TRANSLATIONSDIR}" ) While the motivating case is Qt-specific, the function implementation is being somewhat future proofed for the not-yet created public SBOM API. The new API supports adding files to any target that is backed by an SBOM package, so all targets created by qt_internal_add_module() and friends, as well as ones created by qt_internal_add_sbom(). It can be called multiple times for the same target, with a different set of files, to e.g. assign a different license, or file type per file set. Note that the file set doesn't have anything to do with CMake's concept of file sets. The function is also multi-config aware, and allows specifying different install paths per config, as well as generator expressions in file names. But the multi-config support is a bit wonky, and might need some rethinking in the future. Note that the custom files must be installed and available in the specified qt install path, because the file contents will be checksummed at install time and embedded into the sbom document. Calling the new API does not do installation itself. Implementation wise, the function call flow is - project calls qt_internal_sbom_add_files() one or more times - at finalization time, the _qt_internal_sbom_add_target finalizer is called for a target, which then calls _qt_internal_sbom_handle_target_custom_files() - the latter calls _qt_internal_sbom_handle_target_custom_file_set() for each file set that was added to the target - the latter calls _qt_internal_sbom_handle_multi_config_custom_file() for each input file in the file set, which ultimately calls _qt_internal_sbom_add_custom_file() Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Task-number: QTBUG-128320 Change-Id: Iafde26ebd68f4168b49e55fbc8ad1c251e98d4b0 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-11-18 17:12:17 +01:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
if(is_qt_entity_type)
set(namespaced_target "${QT_CMAKE_EXPORT_NAMESPACE}::${target}")
set(namespaced_private_target "${QT_CMAKE_EXPORT_NAMESPACE}::${target}Private")
set(versionless_target "Qt::${target}")
set(versionless_private_target "Qt::${target}Private")
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
list(APPEND target_names
namespaced_target
namespaced_private_target
versionless_target
versionless_private_target
)
CMake: Rework SBOM PURL handling to handle multiple values Using a single PURL for a target is not enough to represent all the information that an SBOM processing tool might want to know about. For example for the bundled harfbuzz package, we want to inform at least about two versions: the upstream version that was originally imported into Qt sources, and the Qt version or sha1 of the sources that were used, because the 3rdparty code might have been modified and there might be new Qt-specific vulnerabilities. To handle this, we need to generate multiple PURLs for a single target. Introduce six new options to be understood by qt_internal_add_module and other SBOM functions: - PURL_QT_ARGS - PURL_MIRROR_ARGS - PURL_3RDPARTY_UPSTREAM_ARGS - PURL_QT_VALUE - PURL_3RDPARTY_UPSTREAM_VALUE - PURL_MIRROR_VALUE The first three options take multiple arguments, and will generate a PURL for each used variant, based on the purl parsing options that were previously handled. For example passing: PURL_3RDPARTY_UPSTREAM_ARGS PURL_TYPE "github" PURL_NAMESPACE "harfbuzz" PURL_NAME "harfbuzz" PURL_VERSION "v8.5.0" # tag will generate a PURL pointing to the upstream harfbuzz repo hosted on github. The next three options allow specifying a purl value directly as a single string, rather than as separate parts. This might be useful when the PURL is pre-constructed and read from a qt_attribution file. Example: PURL_3RDPARTY_UPSTREAM_VALUE "pkg:github/harfbuzz/harfbuzz@v8.5.0" When no arguments are specified, targets like Qt modules or Qt 3rd party libraries will have automatically generated QT and MIRROR variant PURLs that point to code.qt.io and github.com, along with important info like the version and subdir source path for a given target. Third party libraries are expected to be manually annotated with a 3RDPARTY_UPSTREAM variant PURL that points to the original upstream. In a future change, these will be read from a qt_attribution.json file. The final set of generated PURLs for the harfbuzz package might look like: pkg:github/harfbuzz/harfbuzz@v8.5.0 pkg:github/qt/qtbase@5018b71e99f?library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng pkg:generic/TheQtCompany/qtbase-BundledHarfbuzz@5018b71e99f?vcs_url=https://code.qt.io/qt/qtbase.git@5018b71e99f&library_name=BundledHarfbuzz#src/3rdparty/harfbuzz-ng Additionally a few more purl parsing options are added. Add a PURL_USE_PACKAGE_VERSION option that will use the qt_attribution.json or custom PACKAGE_VERSION value as the PURL version, so it doesn't have to be manually specified. This is an opt-in, and not the default, because some attribution files contain plain text, white-space separated, strings as the version value (like the 'sha3' 3rd party lib) which ends up generating a broken PURL and a failing JSON conversion of the SBOM. So we have to manually annotate targets that should use the attribution json package version, until a better way to handle this can be found. Add a PURL_VCS_URL option that will generate a PURL qualifier (a HTTP query parameter) with the given value, to indicate to SBOM-processing tools what is the upstream repo URL for the package. They can use this information to display known vulnerabilities for the package hosted at that URL. This is mostly useful for the generic QT purl variant, and is automatically generated for Qt entity types. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: Ie000b01b478bef4bff6f4803dd39e37b7a8055d5 Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-24 17:29:11 +02:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
foreach(target_name IN LISTS ${target_names})
if(TARGET "${target_name}")
_qt_internal_sbom_save_spdx_id_for_target("${target_name}" "${package_spdx_id}")
endif()
endforeach()
CMake: Improve SBOM PURL handling Tools that consume SBOMs expect that each SBOM package will contain a unique purl: https://github.com/package-url/purl-spec Each Qt target maps to an SBOM package, so generate a target-specific PURL for each Qt target, by using the combination of the repo name and target name as the purl name. The purl external reference will then look something like: ExternalRef: PACKAGE-MANAGER purl pkg:/TheQtCompany/qtbase-Gui@6.8.0 Also allow customizing the purl for each target by specifying one of the following options to functions like qt_internal_add_module or qt_internal_extend_sbom: - PURL_TYPE - PURL_NAMESPACE - PURL_NAME - PURL_VERSION - PURL_SUBPATH - PURL_QUALIFIERS - NO_PURL - NO_DEFAULT_QT_PURL Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I6926dd773a0ef6fc688664bcac7b23483ecbabe6 Reviewed-by: Kai Köhne <kai.koehne@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-07-22 17:26:29 +02:00
endfunction()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Retrieves a saved spdx id from the target. Might be empty.
function(_qt_internal_sbom_get_spdx_id_for_target target out_var)
get_target_property(spdx_id ${target} _qt_sbom_spdx_id)
set(${out_var} "${spdx_id}" PARENT_SCOPE)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endfunction()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Returns a package infix for a given target sbom type to be used in spdx package id generation.
function(_qt_internal_sbom_get_package_infix type out_infix)
if(type STREQUAL "QT_MODULE")
set(package_infix "qt-module")
elseif(type STREQUAL "QT_PLUGIN")
set(package_infix "qt-plugin")
elseif(type STREQUAL "QML_PLUGIN")
set(package_infix "qt-qml-plugin") # not used at the moment
elseif(type STREQUAL "QT_TOOL")
set(package_infix "qt-tool")
elseif(type STREQUAL "QT_APP")
set(package_infix "qt-app")
elseif(type STREQUAL "QT_THIRD_PARTY_MODULE")
set(package_infix "qt-bundled-3rdparty-module")
elseif(type STREQUAL "QT_THIRD_PARTY_SOURCES")
set(package_infix "qt-3rdparty-sources")
elseif(type STREQUAL "QT_TRANSLATIONS")
set(package_infix "qt-translation")
elseif(type STREQUAL "QT_RESOURCES")
set(package_infix "qt-resource")
elseif(type STREQUAL "QT_CUSTOM")
set(package_infix "qt-custom")
elseif(type STREQUAL "QT_CUSTOM_NO_INFIX")
set(package_infix "qt")
elseif(type STREQUAL "SYSTEM_LIBRARY")
set(package_infix "system-3rdparty")
elseif(type STREQUAL "EXECUTABLE")
set(package_infix "executable")
elseif(type STREQUAL "LIBRARY")
set(package_infix "library")
elseif(type STREQUAL "THIRD_PARTY_LIBRARY")
set(package_infix "3rdparty-library")
elseif(type STREQUAL "THIRD_PARTY_LIBRARY_WITH_FILES")
set(package_infix "3rdparty-library-with-files")
CMake: Add a new SBOM entity type: THIRD_PARTY_SOURCES It mirrors the Qt-specific QT_THIRD_PARTY_SOURCES one, to be used when creating an SBOM attribution target that references a 3rd party library's sources, as opposed to a complete 3rd party library. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I8f7f9f7386ffdc18dd8ae6ee32e39019639f5303 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-02-24 18:35:30 +01:00
elseif(type STREQUAL "THIRD_PARTY_SOURCES")
set(package_infix "3rdparty-sources")
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
elseif(type STREQUAL "TRANSLATIONS")
set(package_infix "translations")
elseif(type STREQUAL "RESOURCES")
set(package_infix "resource")
elseif(type STREQUAL "CUSTOM")
set(package_infix "custom")
elseif(type STREQUAL "CUSTOM_NO_INFIX")
set(package_infix "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
else()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
message(DEBUG "No package infix due to unknown type: ${type}")
set(package_infix "")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(${out_infix} "${package_infix}" PARENT_SCOPE)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endfunction()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
# Returns a package purpose for a given target sbom type.
function(_qt_internal_sbom_get_package_purpose type out_purpose)
if(type STREQUAL "QT_MODULE")
set(package_purpose "LIBRARY")
elseif(type STREQUAL "QT_PLUGIN")
set(package_purpose "LIBRARY")
elseif(type STREQUAL "QML_PLUGIN")
set(package_purpose "LIBRARY")
elseif(type STREQUAL "QT_TOOL")
set(package_purpose "APPLICATION")
elseif(type STREQUAL "QT_APP")
set(package_purpose "APPLICATION")
elseif(type STREQUAL "QT_THIRD_PARTY_MODULE")
set(package_purpose "LIBRARY")
elseif(type STREQUAL "QT_THIRD_PARTY_SOURCES")
set(package_purpose "LIBRARY")
elseif(type STREQUAL "QT_TRANSLATIONS")
set(package_purpose "OTHER")
elseif(type STREQUAL "QT_RESOURCES")
set(package_purpose "OTHER")
elseif(type STREQUAL "QT_CUSTOM")
set(package_purpose "OTHER")
elseif(type STREQUAL "QT_CUSTOM_NO_INFIX")
set(package_purpose "OTHER")
elseif(type STREQUAL "SYSTEM_LIBRARY")
set(package_purpose "LIBRARY")
elseif(type STREQUAL "EXECUTABLE")
set(package_purpose "APPLICATION")
elseif(type STREQUAL "LIBRARY")
set(package_purpose "LIBRARY")
elseif(type STREQUAL "THIRD_PARTY_LIBRARY")
set(package_purpose "LIBRARY")
elseif(type STREQUAL "THIRD_PARTY_LIBRARY_WITH_FILES")
set(package_purpose "LIBRARY")
CMake: Add a new SBOM entity type: THIRD_PARTY_SOURCES It mirrors the Qt-specific QT_THIRD_PARTY_SOURCES one, to be used when creating an SBOM attribution target that references a 3rd party library's sources, as opposed to a complete 3rd party library. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I8f7f9f7386ffdc18dd8ae6ee32e39019639f5303 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-02-24 18:35:30 +01:00
elseif(type STREQUAL "THIRD_PARTY_SOURCES")
set(package_purpose "LIBRARY")
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
elseif(type STREQUAL "TRANSLATIONS")
set(package_purpose "OTHER")
elseif(type STREQUAL "RESOURCES")
set(package_purpose "OTHER")
elseif(type STREQUAL "CUSTOM")
set(package_purpose "OTHER")
elseif(type STREQUAL "CUSTOM_NO_INFIX")
set(package_purpose "OTHER")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
else()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(package_purpose "OTHER")
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endif()
CMake: Split SBOM implementation into separate files The SBOM implementation got somewhat large. Split the code into several new QtPublicSbomFooHelpers.cmake files, to make it more manageable. No code or behavior was changed. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: Ia0ca1792eec21d12c4bb4cabe63279e1f5c07e3d Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 12:45:31 +01:00
set(${out_purpose} "${package_purpose}" PARENT_SCOPE)
CMake: Generate an SPDX v2.3 SBOM file for each built repository This change adds a new -sbom configure option to allow generating and installing an SPDX v2.3 SBOM file when building a qt repo. The -sbom-dir option can be used to configure the location where each repo sbom file will be installed. By default it is installed into $prefix/$archdatadir/sbom/$sbom_lower_project_name.sdpx which is basically ~/Qt/sbom/qtbase-6.8.0.spdx The file is installed as part of the default installation rules, but it can also be installed manually using the "sbom" installation component, or "sbom_$lower_project_name" in a top-level build. For example: cmake install . --component sbom_qtbase CMake 3.19+ is needed to read the qt_attribution.json files for copyrights, license info, etc. When using an older cmake version, configuration will error out. It is possible to opt into using an older cmake version, but the generated sbom will lack all the attribution file information. Using an older cmake version is untested and not officially supported. Implementation notes. The bulk of the implementation is split into 4 new files: - QtPublicSbomHelpers.cmake - for Qt-specific collecting, processing and dispatching the generation of various pieces of the SBOM document e.g. a SDPX package associated with a target like Core, a SDPX file entry for each target binary file (per-config shared library, archive, executable, etc) - QtPublicSbomGenerationHelpers.cmake - for non-Qt specific implementation of SPDX generation. This also has some code that was taken from the cmake-sbom 3rd party project, so it is dual licensed under the usual Qt build system BSD license, as well as the MIT license of the 3rd party project - QtPublicGitHelpers.cmake - for git related features, mainly to embed queried hashes or tags into version strings, is dual-licensed for the same reasons as QtPublicSbomGenerationHelpers.cmake - QtSbomHelpers.cmake - Qt-specific functions that just forward arguments to the public functions. These are meant to be used in our Qt CMakeLists.txt instead of the public _qt_internal_add_sbom ones for naming consistency. These function would mostly be used to annotate 3rd party libraries with sbom info and to add sbom info for unusual target setups (like the Bootstrap library), because most of the handling is already done automatically via qt_internal_add_module/plugin/etc. The files are put into Public cmake files, with the future hope of making this available to user projects in some capacity. The distinction of Qt-specific and non-Qt specific code might blur a bit, and thus the separation across files might not always be consistent, but it was best effort. The main purpose of the code is to collect various information about targets and their relationships and generate equivalent SPDX info. Collection is currently done for the following targets: Qt modules, plugins, apps, tools, system libraries, bundled 3rd party libraries and partial 3rd party sources compiled directly as part of Qt targets. Each target has an equivalent SPDX package generated with information like version, license, copyright, CPE (common vulnerability identifier), files that belong to the package, and relationships on other SPDX packages (associated cmake targets), mostly gathered from direct linking dependencies. Each package might also contain files, e.g. libQt6Core.so for the Core target. Each file also has info like license id, copyrights, but also the list of source files that were used to generate the file and a sha1 checksum. SPDX documents can also refer to packages in other SPDX documents, and those are referred to via external document references. This is the case when building qtdeclarative and we refer to Core. For qt provided targets, we have complete information regarding licenses, and copyrights. For bundled 3rd party libraries, we should also have most information, which is usually parsed from the src/3rdparty/libfoo/qt_attribution.json files. If there are multiple attribution files, or if the files have multiple entries, we create a separate SBOM package for each of those entries, because each might have a separate copyright or version, and an sbom package can have only one version (although many copyrights). For system libraries we usually lack the information because we don't have attribution files for Find scripts. So the info needs to be manually annotated via arguments to the sbom function calls, or the FindFoo.cmake scripts expose that information in some form and we can query it. There are also corner cases like 3rdparty sources being directly included in a Qt library, like the m4dc files for Gui, or PCRE2 for Bootstrap. Or QtWebEngine libraries (either Qt bundled or Chromium bundled or system libraries) which get linked in by GN instead of CMake, so there are no direct targets for them. The information for these need to be annotated manually as well. There is also a distinction to be made for static Qt builds (or any static Qt library in a shared build), where the system libraries found during the Qt build might not be the same that are linked into the final user application or library. The actual generation of the SBOM is done by file(GENERATE)-ing one .cmake file for each target, file, external ref, etc, which will be included in a top-level cmake script. The top-level cmake script will run through each included file, to append to a "staging" spdx file, which will then be used in a configure_file() call to replace some final variables, like embedding a file checksum. There are install rules to generate a complete SBOM during installation, and an optional 'sbom' custom target that allows building an incomplete SBOM during the build step. The build target is just for convenience and faster development iteration time. It is incomplete because it is missing the installed file SHA1 checksums and the document verification code (the sha1 of all sha1s). We can't compute those during the build before the files are actually installed. A complete SBOM can only be achieved at installation time. The install script will include all the generated helper files, but also set some additional variables to ensure checksumming happens, and also handle multi-config installation, among other small things. For multi-config builds, CMake doesn't offer a way to run code after all configs are installed, because they might not always be installed, someone might choose to install just Release. To handle that, we rely on ninja installing each config sequentially (because ninja places the install rules into the 'console' pool which runs one task at a time). For each installed config we create a config-specific marker file. Once all marker files are present, whichever config ends up being installed as the last one, we run the sbom generation once, and then delete all marker files. There are a few internal variables that can be set during configuration to enable various checks (and other features) on the generated spdx files: - QT_INTERNAL_SBOM_VERIFY - QT_INTERNAL_SBOM_AUDIT - QT_INTERNAL_SBOM_AUDIT_NO_ERROR - QT_INTERNAL_SBOM_GENERATE_JSON - QT_INTERNAL_SBOM_SHOW_TABLE - QT_INTERNAL_SBOM_DEFAULT_CHECKS These use 3rd party python tools, so they are not enabled by default. If enabled, they run at installation time after the sbom is installed. We will hopefully enable them in CI. Overall, the code is still a bit messy in a few places, due to time constraints, but can be improved later. Some possible TODOs for the future: - Do we need to handle 3rd party libs linked into a Qt static library in a Qt shared build, where the Qt static lib is not installed, but linked into a Qt shared library, somehow specially? We can record a package for it, but we can't create a spdx file record for it (and associated source relationships) because we don't install the file, and spdx requires the file to be installed and checksummed. Perhaps we can consider adding some free-form text snippet to the package itself? - Do we want to add parsing of .cpp source files for Copyrights, to embed them into the packages? This will likely slow down configuration quite a bit. - Currently sbom info attached to WrapFoo packages in one repo is not exported / available in other repos. E.g. If we annotate WrapZLIB in qtbase with CPE_VENDOR zlib, this info will not be available when looking up WrapZLIB in qtimageformats. This is because they are IMPORTED libraries, and are not exported. We might want to record this info in the future. [ChangeLog][Build System] A new -sbom configure option can be used to generate and install a SPDX SBOM (Software Bill of Materials) file for each built Qt repository. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I9c730a6bbc47e02ce1836fccf00a14ec8eb1a5f4 Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io> Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2024-03-07 18:02:56 +01:00
endfunction()
CMake: Fix sbom git vars not being available in various scopes Initially the git vars were assigned to the parent scope of the _qt_internal_sbom_begin_project function, with the intent to set them in the global scope. But the function was later wrapped in other functions, so the variables stopped being accessible. Instead of playing with recursive PARENT_SCOPEs, save the variables in global properties like we do for other info, and use a new _qt_internal_sbom_get_git_version_vars() function to query the vars in the code that needs them. This fixes generated purls to contain the git version and hashes. Also add a new internal API wrapper macro called qt_internal_sbom_get_git_version_vars to allow calling it in other repos. Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I061b34f418c1ecc1c66c8c01ef758d2f40611ede Reviewed-by: Alexey Edelev <alexey.edelev@qt.io>
2025-01-10 11:46:36 +01:00
# Queries the current project git version variables and sets them in the parent scope.
function(_qt_internal_sbom_get_git_version_vars)
get_cmake_property(QT_SBOM_GIT_VERSION QT_SBOM_GIT_VERSION)
get_cmake_property(QT_SBOM_GIT_VERSION_PATH QT_SBOM_GIT_VERSION_PATH)
get_cmake_property(QT_SBOM_GIT_HASH QT_SBOM_GIT_HASH)
get_cmake_property(QT_SBOM_GIT_HASH_SHORT QT_SBOM_GIT_HASH_SHORT)
set(QT_SBOM_GIT_VERSION "${QT_SBOM_GIT_VERSION}" PARENT_SCOPE)
set(QT_SBOM_GIT_VERSION_PATH "${QT_SBOM_GIT_VERSION_PATH}" PARENT_SCOPE)
set(QT_SBOM_GIT_HASH "${QT_SBOM_GIT_HASH}" PARENT_SCOPE)
set(QT_SBOM_GIT_HASH_SHORT "${QT_SBOM_GIT_HASH_SHORT}" PARENT_SCOPE)
endfunction()
CMake: When available add the repo configure line to the SBOM When Qt is configured using the configure script, rather than directly via CMake, we can obtain the full list of configure arguments passed from the config.opt file. Add the configure line to the repo project SBOM package comment field when it is available. Pick-to: 6.8 Task-number: QTBUG-122899 Change-Id: I0c5554dcda2f71ec4ba034b14c82a99757dc790c Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2024-10-17 17:20:29 +02:00
# Returns the configure line used to configure the current repo or top-level build, by reading
# the config.opt file that the configure script writes out.
# Returns an empty string if configure was not called, but CMake was called directly.
# If the build is reconfigured with bare CMake, the config.opt remains untouched, and thus
# the previous contents is returned.
function(_qt_internal_get_configure_line out_var)
set(content "")
if(QT_SUPERBUILD OR PROJECT_NAME STREQUAL "QtBase")
set(configure_script_name "qt6/configure")
elseif(PROJECT_NAME STREQUAL "QtBase")
set(configure_script_name "qtbase/configure")
else()
_qt_internal_sbom_get_root_project_name_lower_case(repo_project_name_lowercase)
set(configure_script_name "qt-configure-module <sources>/${repo_project_name_lowercase}")
endif()
if(QT_SUPERBUILD)
set(config_opt_path "${PROJECT_BINARY_DIR}/../config.opt")
else()
set(config_opt_path "${PROJECT_BINARY_DIR}/config.opt")
endif()
if(NOT EXISTS "${config_opt_path}")
message(DEBUG "Couldn't find config.opt file in ${config_opt} for argument extraction.")
set(${out_var} "${content}" PARENT_SCOPE)
return()
endif()
file(STRINGS "${config_opt_path}" args)
list(JOIN args " " joined_args)
set(content "${configure_script_name} ${joined_args}")
string(STRIP "${content}" content)
set(${out_var} "${content}" PARENT_SCOPE)
endfunction()
CMake: Add a few more internal sbom helper functions Add a few internal functions to allow - getting the sbom project supplier - project supplier url - project namespace - computing a project namespace - computing a project file name - getting a sanitized spdx id given a hint Pick-to: 6.8 6.9 Task-number: QTBUG-122899 Change-Id: I0dc3df274eaf6882a6af021aabee75501b5083f8 Reviewed-by: Moss Heim <moss.heim@qt.io> Reviewed-by: Joerg Bornemann <joerg.bornemann@qt.io>
2025-01-21 19:32:35 +01:00
function(_qt_internal_sbom_compute_project_namespace out_var)
set(opt_args "")
set(single_args
SUPPLIER_URL
PROJECT_NAME
VERSION_SUFFIX
)
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
if(NOT arg_PROJECT_NAME)
message(FATAL_ERROR "PROJECT_NAME must be set")
endif()
if(NOT arg_SUPPLIER_URL)
message(FATAL_ERROR "SUPPLIER_URL must be set")
endif()
string(TOLOWER "${arg_PROJECT_NAME}" project_name_lowercase)
set(version_suffix "")
if(arg_VERSION_SUFFIX)
set(version_suffix "-${arg_VERSION_SUFFIX}")
else()
_qt_internal_sbom_get_git_version_vars()
if(QT_SBOM_GIT_VERSION)
set(version_suffix "-${QT_SBOM_GIT_VERSION}")
endif()
endif()
# Used in external refs, it should be either aa URI + UUID or a URI + checksum.
# We currently use a URI + git version, which is probably not conformant to the spec.
set(repo_name_and_version "${project_name_lowercase}${version_suffix}")
set(repo_spdx_namespace
"${arg_SUPPLIER_URL}/spdxdocs/${repo_name_and_version}")
set(${out_var} "${repo_spdx_namespace}" PARENT_SCOPE)
endfunction()
function(_qt_internal_sbom_compute_project_file_name out_var)
set(opt_args
EXTENSION_JSON
)
set(single_args
PROJECT_NAME
VERSION_SUFFIX
)
set(multi_args "")
cmake_parse_arguments(PARSE_ARGV 1 arg "${opt_args}" "${single_args}" "${multi_args}")
_qt_internal_validate_all_args_are_parsed(arg)
if(NOT arg_PROJECT_NAME)
message(FATAL_ERROR "PROJECT_NAME must be set")
endif()
string(TOLOWER "${arg_PROJECT_NAME}" project_name_lowercase)
set(version_suffix "")
if(arg_VERSION_SUFFIX)
set(version_suffix "-${arg_VERSION_SUFFIX}")
elseif(QT_REPO_MODULE_VERSION)
set(version_suffix "-${QT_REPO_MODULE_VERSION}")
endif()
if(arg_EXTENSION_JSON)
set(extension "spdx.json")
else()
set(extension "spdx")
endif()
set(result
"${project_name_lowercase}${version_suffix}.${extension}")
set(${out_var} "${result}" PARENT_SCOPE)
endfunction()
Reference in New Issue Copy Permalink