Automated digital signing of OS artifacts #

• Date proposed: 2025-06-17
• RFC MR: https://gitlab.archlinux.org/archlinux/rfcs/-/merge_requests/59

Summary #

Introduce a centralized, hardware backed solution for the digital signing of OS artifacts. Gradually replace the need for manual signing of artifacts throughout the distribution.

The stepwise plan in this document will eventually lead to changes for the following existing roles within Arch Linux staff:

• Package maintainers will no longer sign packages using their individual OpenPGP private key.
• The amount of OpenPGP certificates for main signing key holders to care for will be drastically reduced.
• The DevOps team will have to monitor and administrate additional physical machines.

New groups of people within Arch Linux staff will

• collectively take care of the administrative credentials for the described system as holders of shares of a shared secret,
• provide software upgrades for components of the system as developers of Signstar
• and create releases for a central, image-based OS as developers of Signstar OS.

For details refer to the section changes for users and staff.

Motivation #

Arch Linux has relied on digital signatures for various artifacts over a number of years now. At the time of writing, users rely on OpenPGP signatures and delegated authentication based on a set of distribution-specific trust anchors when verifying the authenticity and integrity of package files and installation media.

In this setup, package maintainers each use a dedicated OpenPGP private key to issue signatures for the artifacts they built. Users rely on a dedicated set of OpenPGP certificates and third-party signatures provided by archlinux-keyring to delegate trust from the distribution-specific trust anchors to a specific certificate of a package maintainer when verifying an artifact signature.

Unfortunately, the approach of tracking the certificates of so many individuals and using them in this verification scheme comes at a disadvantage:

• Handling certificates in archlinux-keyring entails a certain amount of administrative overhead for each certificate when adding, changing and decommissioning them.
• Arch Linux can not vouch for the operational security of each of its package maintainers. Instead of hardware-backed private keys, a software key may be used. Individuals may be compromised without their knowledge and their private key material may be used to compromise packages (either for all users or for targeted groups).
• The release of artifacts such as the archiso-based installation medium and the arch-boxes-based virtual machine images are currently not securely automatable, as the private key material could be exfiltrated from Arch Linux’s continuous integration setup.
• All package maintainers have access to the central package repository server, which employs the dbscripts repository management software for managing the official package repositories. Although technically possible, repository sync databases are currently not signed, as signing would need to take place on this central host, which is a security risk. On one hand, when relying on the OpenPGP private key of individual packagers, signing operations would need to be forwarded to the host and changes to a repository sync database would need to be signed by individual packagers. Changes to the repository sync database take place when packages are added, removed or updated in a given repository. On the other hand, if a dedicated key was to be deployed for the host it could easily be abused for malicious package updates.
• The Arch Linux Package Management tools pacman-key and pacman integrate with a system-wide, stateful GnuPG keyring which is populated by the data accumulated in archlinux-keyring. This technological integration tends to be brittle for users due to the handling of certificate data outside of the package installation process. This problem is amplified by having many certificates, and certificates with moving windows of validity (see expiration).

Moving to a central, transparent and hardware-backed signing solution addresses the above points by

• reducing the amount of active signing keys to a small, purpose driven set of keys without expiration,
• rendering the exfiltration of private key material unfeasible or impossible by not relying on software keys,
• enabling the automation of digital signing in continuous integration pipelines by granting access to dedicated signing operations,
• allowing to further lock down the central package repository server in the future and to sign repository metadata automatically,
• and alleviating end-user issues around package verification by introducing long-lived package signing keys.

For this purpose several solutions and approaches have been evaluated, that allow for secure, future-proof deployment, driven by open source software.

Specification #

The Signstar project has been started to establish signing automation for Arch Linux. In the course of this work, a set of requirements has been established:

• An auditable, extensible and transparent Hardware Security Module (HSM), based on free/open-source software is used (with ICANN’s effort of switching HSMs in mind).
• A central physical host, with a dedicated, minimal read-only operating system that allows to transparently manage a connected HSM is used to digitally sign artifacts using protocol-specific framing (e.g. OpenPGP).
• Following the principle of least privilege, administrative credentials for the HSM are temporarily created on the central host and are then managed using shared secrets (see Shamir’s Secret Sharing) held by dedicated individuals. Non-administrative credentials for the HSM are generated as needed on the central host and are stored in locations only accessible by dedicated, locked down login users that can not expose those credentials.
• A dedicated threat model (see design document) describes the issues that are specifically in scope for the system, as well as those that are out of scope.
• Clients request and receive signatures over authenticated channels to the central host and have no direct access or credentials to the HSM.
• Apart from login users for non-administrative workflows towards the HSM (i.e. requesting signatures, requesting certificates, requesting backups) no other users have access to the central host during runtime.
• Client and host components are written in a memory-safe and robust programming language and are extensively documented.
• A dedicated wire format for issuing signing requests and receiving signatures is used over a secure channel between client and central host (see design document).

HSM #

After an initial research period, the Nitrokey NetHSM has been considered as HSM, as it provides a formally verified, open-source operating system (see the NetHSM design document for further details), as well as a containerized environment for integration testing. A dedicated security audit by Secfault Security has been issued for the device and its code in 2020/2021.

Similar to other enterprise hardware solutions the Nitrokey NetHSM requires colocation for the device, which also includes a custom machine to front it.

Due to cost and maintenance considerations, a simpler and slower approach is chosen instead, which facilitates a USB HSM that is compatible with the current dedicated hosting provider used by Arch Linux.

Signstar #

The set of Signstar tools is written in the Rust programming language to circumvent memory safety issues and rely on a modern systems programming language for speed and ease of development.

A dedicated, image-based operating system based on Arch Linux - Signstar OS - is created using mkosi and provides an A/B boot based approach, that is deployed and updated using operating system images. A set of Signstar OS developers maintains a dedicated keyring for the centrally built system image releases and issues necessary digital signatures for them.

Signstar OS is run on a dedicated host, referred to as Signstar host, and provides a mostly read-only environment that is altered by updating it. The only read-writable location on the system is encrypted using the host’s Trusted Platform Module (TPM 2.0) and solely contains credentials for non-administrative users.

System overview #

Clients connect to a Signstar host over SSH, which provides the following advantages:

• Reduced complexity by not having to implement a dedicated authentication stack.
• Well tested, understood and integrated into the Linux operating system.
• Private key material used for authentication on a host can be guarded by various hardware backed methods, such as TPM 2.0, smartcards, etc.
• Each login user can be assigned a specific executable, based on their role.
• All successful connections to the Signstar host are logged and forwarded to a central host for log aggregation. These logs include details about the specific login users, their used credentials, the current version of the host’s configuration and the reason for the connection (e.g. the hashes for a signing operation and an alleged file name).

The following diagram showcases a signing workflow based on Signstar components on a high level.

  sequenceDiagram
    participant C as Client
    participant S as Signstar
    participant H as HSM

    Note over S: Signstar credentials
    Note over H: HSM credentials

    S ->> H: HSM is configured using *signstar-configure*
    C ->>+ S: User "A" requests signature using *signstar-request-signature*
    S ->> S: Host user "A" is mapped to HSM operator user "X" by *signstar-sign*
    S ->> H: Signature is requested using operator user "X" by *signstar-sign*
    H ->> S: Raw cryptographic signature is received by *signstar-sign*
    S ->>- C: Signature for user "A" is returned by *signstar-sign*

The above diagram outlines the workflow of one client connecting to one Signstar host. For redundancy reasons, there should in reality be more than one Signstar host and HSM backend in the future. Each client is supposed to request signatures from one of the Signstar hosts in a round-robin fashion.

In a set of Signstar systems (each consisting of a Signstar host and an HSM backend) redundancy of the cryptographic material in the HSM backends is achieved by creating a backup for one system and restoring the other system with it. The result is a set of Signstar systems with identical cryptographic material in their respective HSM backends.

Digital signatures #

At the time of writing, the Signstar project focuses on supporting OpenPGP, as it is the dominating cryptographic protocol used in Arch Linux. However, the integration with the lower-level cryptographic operations of the HSM as backend allows for extensibility and the support for further technologies such as SSH and X.509 in the future.

Dedicated raw cryptographic keys for signing are created on the HSM backend. OpenPGP certificates associated with each raw cryptographic key are also held on the backend and can be downloaded. These certificates are added to the data held in the archlinux-keyring project and (depending on their use-case) trust is delegated to them from the distribution-specific trust anchors via third-party signatures.

Roadmap #

The use of Signstar is rolled out in several steps. This is mainly to ensure the quality and security of the system before actual use and to allow to accommodate for the various use-cases while ironing out any issues that may arise on the way.

Virtual test system #

The first milestone involves deploying a Signstar host as a virtual machine, which operates against a virtual HSM environment as backend. This allows to verify all functional properties of the system while signing dummy artifacts using dummy keys. Further, the various assumptions about the use of client integration towards a Signstar host can be validated.

A virtual test system will remain operative all throughout the use of the Signstar system to allow for tests of system upgrade, system roll back and initial deployment scenarios.

This system is considered a foundational learning ground to iron out any deployment issues and to improve existing documentation.

Write manual on deployment and operation #

Good documentation for the deployment and ongoing operation of Signstar is essential to its success.

Before an actual hardware-backed system is deployed a dedicated deployment and operational manual is created.

Any potential issues identified in the virtual test system are incorporated iteratively. This process should entail close collaboration with and feedback rounds from the DevOps team. It should cover details from initial bootstrap, over updates and schemes for digital signatures for images of Signstar OS to disaster recovery of the entire Signstar system.

Acquire dedicated hardware #

After the initial virtual test system has been running successfully through various test scenarios and the initial manual on deployment and operation is finished and validated with DevOps, at least two physical HSMs need to be acquired.

To establish a Signstar host for an HSM backend, a dedicated machine with integrated or discrete TPM 2.0 and remote management access support needs to be used.

Hardware test system #

To extend and refine the manual on deployment and operation an initial hardware based test system needs to be created.

This allows for discrepancies between virtual abstraction and hardware-based setup to be detected and documented.

Any integration towards this hardware test system is only used in a testing capacity (e.g. any created signatures are only valid in the context of the given test system but not for Arch Linux in general).

Security audit #

Before production deployment of the Signstar system, ideally an audit by third-party security researchers should take place.

Such an audit should encompass gray-box testing of the entire hardware test system and its update mechanism, as well as a code audit of the Signstar code base. Security audits should be carried out by professional security labs with a public track record and an established understanding of the underlying technology and involved programming languages.

As the costs of such an audit are estimated at over €100.000, a sponsored code audit should be applied for. A missing audit should not circumvent deployment, but instead be sought after in the future.

Production system #

A Signstar system, consisting of a Signstar host and one HSM backend each should be hosted redundantly in a data center. Redundant hosting requires to have at least two Signstar systems, ideally in two separate locations.

Sign low impact artifacts #

There are several contexts in which signatures are created manually or using unsecured private key material. These include signatures for installation media and virtual machines. Signing these artifacts is considered a low impact action, because the current workflow is either manual or unsecured, happens on a low cadence and adding a drop-in signature by an individual of the Arch Linux team would be possible. Further, end-user systems can not be broken or compromised by addressing these artifacts and delegated authentication for the signatures is currently only partly advertised.

For these artifacts, the signing should take place on dedicated hosts, which guard their SSH keys using the host’s integrated or discrete TPM 2.0. Such a host is then used specifically in the CI pipeline of the projects building the installation media and virtual machines. Each of these hosts rely on Signstar client software to issue signing requests and receive signatures.

Sign medium impact artifacts #

Currently, Arch Linux does not provide digital signatures for repository sync databases (aka repository metadata). However, doing so would improve the supply chain security for all users of the distribution. Signing the repository sync databases is considered a medium impact action, as users would initially need to actively opt-in to verify them.

The automatic signing of repository sync databases may involve creating a dedicated user on the central package repository server to decouple the package maintainer’s login user from the user with access to the Signstar ssh credentials. Here again, the host should guard the SSH key used for connecting to the Signstar host using the host’s integrated or discrete TPM 2.0. In addition to user separation, refactoring and re-architecturing of parts of the dbscripts project are likely necessary. Here, the logins of all package maintainers and administrators must be logged, so that in case of malicious activity, all signature requests can be pin-pointed to the responsible individual and cross-referenced with the dedicated logs of the Signstar host.

Once signing of repository sync databases is automated and works reliably, the distribution-wide default should be adapted so that end-user systems require valid signatures for them.

Sign high impact artifacts #

All package files are currently signed by individual package maintainers. Automating the signing of package files implies, that an automated, central build system provides a secure way of delegating the signing operation to the Signstar system. Signing the package files is considered a high impact action, because signing these artifacts happens on a high cadence and all Arch Linux users rely on signature verification of package files by default already.

To allow for this target to be met, an automated build system (e.g. buildbtw) must provide secure means of authenticating against the Signstar host. Each host in such a central system should guard its SSH key used for connecting to the Signstar host using the host’s integrated or discrete TPM 2.0.

Once signing of package files is automated and works reliably, the per package maintainer OpenPGP certificates in archlinux-keyring should be decommissioned by revoking the third-party signatures issued by the distribution-specific trust anchors.

Disaster recovery #

Centralized signing needs a carefully laid out disaster recovery plan. This section outlines a selected set of potential issues and their suggested solutions.

Note that, right up until the roadmap is concluded (after being able to sign high impact artifacts), packagers will still be able to sign artifacts manually. For brevity this document therefore intentionally focuses on a high-level view of the topic and only highlights disaster recovery for a fully established system.

Specific procedures should be added to the project documentation of Signstar OS. However, it is advisable to exercise deterministic disaster recovery scenarios on a regular basis. These may include

• the restore from backup of the HSM backend of a Signstar host,
• or the re-provisioning of a Signstar host (depending on feasibility).

Signstar host hardware failure #

If the Signstar host hardware fails, it needs to be replaced by new hardware.

For this to happen, the same or similar hardware should to be acquired. Once a new machine is available, it needs to be bootstrapped with the latest version of Signstar OS.

Once the new Signstar host is available online, the shares of the administrative credentials need to be uploaded again, so that the Signstar host can re-create non-administrative passphrases.

HSM failure #

When an HSM hardware fails, it needs to be replaced by new hardware.

For this to happen, a replacement HSM is ordered from the manufacturer. Once the hardware is available, it is sent to the data center and connected to the idle Signstar host by the hoster.

Once the connection is established, an HSM backup and the shares of the administrative credentials are uploaded to the Signstar host, so that it can restore the new HSM from backup and re-create non-administrative credentials.

Changes for users and staff #

Centrally signing the OS artifacts of Arch Linux has an impact on users and staff, that is outlined in the following sections.

Users #

No manual intervention should be needed for users of Arch Linux. Automated signing and the accompanying certificates will be integrated with existing verification infrastructure.

The only change observable to the users should be that fewer certificates are used for the verification of OS artifacts and that those certificates do not expire.

Package maintainers #

Once an automated build service is rolled out and the automated signing service is able to reliably sign high impact artifacts, package maintainers will not need to sign packages anymore.

Packager maintainers will still be required to maintain a cryptographic key for the authentication towards the established source repositories of the distribution and for the signing of other relevant artifacts (e.g. git tags or release artifacts).

Main signing key holders #

The persons in charge of the distribution-specific trust anchors issue third-party signatures for the OpenPGP certificates of package maintainers. This role remains the same for the OpenPGP certificates maintained by the Signstar system.

However, going forward it is expected that there will only be as many OpenPGP certificates as there are centrally managed build machines. Further, once the automated signing service is able to reliably sign high impact artifacts, the workload for main signing key holders should decrease after the decommissioning (read: removal of trust) of package signing keys held by individual package maintainers.

Holders of shares of shared secrets #

A set of people from Arch Linux staff is required to hold shares of the shared secret (encrypted using Shamir’s Secret Sharing) with which administrative credentials for the Signstar host are encrypted.

Each shareholder is represented by a dedicated entry in the Signstar configuration. It defines the SSH public key to use for authentication towards the Signstar host and the OpenPGP certificate used by the Signstar host to encrypt a share upon download.

Shareholders need to keep their

• SSH public key up-to-date to be able to connect to a Signstar host for uploading and downloading their share,
• and their OpenPGP certificate up-to-date to ensure that they can decrypt shares encrypted by the Signstar host.

Shareholders are expected to keep their shares always encrypted and on a safe storage device (e.g. ideally a drive that is not always connected to an online machine). Further instructions around handling and storage of shares should be added in dedicated documentation when writing the manual on deployment and operation.

A sufficient majority of shares is required when

• a Signstar host is provisioned for the first time,
• a backup is supplied to the Signstar host to restore the state of the HSM backend (this is also required when restoring the state of one Signstar system onto another to maintain the cryptographic material redundancy),
• a new client (e.g. a new build machine) to the Signstar host is created, which requires an update to the Signstar configuration,
• a new user for backup or metrics retrieval purposes is added to the Signstar configuration,
• or the version of the Signstar configuration file is incremented to rotate the administrative credentials.

The exact selection process for the group of shareholders is outside of the scope of this document. However, it should ideally be based on physical distribution around the globe, with a reasonable distribution in different timezones, so that maintenance operations can take place swiftly.

Access to the administrative credentials is guaranteed by the availability of the majority of shares to the shared secret. Once it is clear that this majority can no longer be guaranteed (e.g. due to people leaving the distribution), a new version of the Signstar host configuration must be created, so that administrative credentials are recreated and new shareholders can replace those outside of the trusted active set.

DevOps #

Similar to the other hardware maintained for Arch Linux distribution purposes, the Signstar systems will be maintained by the DevOps team. One distinguishing difference is, that the Signstar host does not offer the traditional interface for access (e.g. “login as privileged user and update the system or alter a configuration file”) as it is an image-based OS. Changes to the system are possible via a central source repository from which an updated OS image is built.

A Signstar host provides access to its own logs and metrics and those of the HSM backend it is connected to with the help of dedicated system users. Logs and metrics of the Signstar systems are forwarded to the central logging and metrics services and need to be monitored for anomalies by the DevOps team.

Developers of Signstar #

People working on the Signstar project need to follow best practices when it comes to the handling of CVEs in the code base.

The release of new versions of Signstar components needs to be streamlined and has to happen in a coordinated and swift way.

Developers of Signstar OS #

A set of people from Arch Linux staff is required to handle changes to Signstar OS, to maintain a dedicated keyring for releases and to create releases for this custom image-based OS.

The developers need to follow best practices when it comes to the handling of CVEs in the components of the dedicated operating system.

Releases of Signstar OS should require digital signatures by at least two individual developers. Each release should be rolled out stepwise to the available Signstar systems. This process needs to be streamlined and has to happen in a coordinated and swift way.

An update to a Signstar host’s Signstar OS is required when

• any change to the Signstar configuration is required (e.g. a new client is added, a new user for backup or metrics retrieval is added, or the administrative credentials are rotated),
• or a CVE has been detected in a component used in Signstar OS, that could lead to the compromise of the Signstar host.

Additionally, it is advisable to keep the Signstar OS deployment generally up to date (e.g. a monthly cadence).

The exact selection process for the group of developers is outside of the scope of this document. However, it should ideally be based on responsibility, physical distribution around the globe, with a reasonable distribution in different timezones, so that releases can take place swiftly.

Total costs #

The total costs of the proposed system are split up into one-time costs, recurring monthly costs and extra costs.

One-time #

Monthly #

Extra #

In case of problem, a KVM console needs to be attached to a host. KVM consoles are provided based on availability are free for the first three hours and cost an additional €8.40 ¹ for an additional three hours.

Future work #

While items below are not on the deployment roadmap, the entire system is designed to accommodate a broader set of use-cases.

Transparency log #

In the future, signatures issued by Signstar should be published in a tamper-proof transparency log (see also Certificate Transparency). Relying on the attestation of a third-party, this allows to maximize the trustworthiness of each signature, by enabling users to verify that a certain signature has been created at a certain point in time.

Currently, if a client requesting a signature from the Signstar host is compromised, it can potentially be used to craft valid signatures for a malicious artifact.

Transparency logs can help to close this gap by providing users a way for checking, if what they are receiving is the same as what everyone else is receiving. However, publishing signature records in a transparency logs is only one half of the solution. For users to benefit, a network of independent monitors must exist that check the log’s consistency (e.g. to ensure that there are no rollbacks). Additionally, the user’s verification must encompass a check for the signature’s inclusion in the log before returning success from the verification function.

Transparency logs are able to provide a third-party view on signing operations. They become particularly useful when the log entries also contain data about the requester and issuer of an operation. With the help of this data detailed introspection is possible in case of compromise of the requester or issuer.

While publishing the signatures in a transparency log is in scope of the Signstar project, the remaining integration is not and needs to be evaluated separately.

As is, the Signstar system will provide central logging functionality for the Signstar host and the HSM backend. This will allow isolating and pin-pointing the source of a potential compromise, but it does not provide the same guarantees as a remote tamper-proof transparency log and adjacent monitoring tools.

Secure Boot #

Arch Linux currently does not maintain a certificate for the signing of Secure Boot payloads.

Securely maintaining and using such a certificate for X.509 signature schemes like PKCS#7/CMS is desirable to allow for inclusion in default certificate lists for hardware vendors. Using such a certificate for the signing of payloads relevant to Secure Boot would enable users of Arch Linux to use Secure Boot enabled systems out of the box.

The HSM offers generic signing primitives, as well as generic certificate storage. It is therefore possible to use this backend for the purpose of signing Secure Boot related payloads.

However, integration with the build process of relevant payloads as well as the packaging system need to be taken into consideration, which needs to be evaluated separately.

Kernel module signing #

The Linux kernel allows for cryptographically signing kernel modules during build time and to check these digital signatures during runtime. This can increase the security of the kernel by only allowing a specific set of modules to be loaded.

Similar to the signing of Secure Boot payloads, kernel modules could be signed with the help of Signstar OS in the future. For this to happen, it could be evaluated to write a PKCS#11 module, which forwards signing requests to a Signstar host, since the Linux kernel build process supports signing using PKCS#11 URIs.

However, this integration needs to be evaluated separately as it requires changes to the kernel package build process and the build tooling.

Drawbacks #

The proposed solution involves an implicitly managed, image-based OS, which is contrary to current approaches on system maintenance from the perspective of Arch Linux DevOps. Maintaining a dedicated set of additional machines means additional effort.

Overall, maintaining the described system requires effort:

• Developers need to ensure the quality and robustness of the Signstar system.
• Deployments and the addition of new clients requires good coordination with the set of cryptographic shareholders.
• The Signstar system may be perceived as opaque by its users, as it differs from the current, decentralized approach of creating digital signatures.

The one-time costs of two Signstar systems, including two backup HSMs are estimated at around €2642. The recurring costs for hosting two Signstar systems are estimated at around €74/month. This is an investment that needs to be covered using Arch Linux’s funds held by SPI.

Unresolved Questions #

Alternatives Considered #

For the Signstar solution, the following hardware scenarios have been evaluated:

During research the following list of legacy alternatives has been considered:

• fero is written in Rust. Unmaintained for more than five years. Uses closed-source YubiHSM2. The Signing payload contains the actual artifact to sign (which in case of packages would be huge). Uses the unmaintained pretty_good crate for OpenPGP.
• sigul is a mix of C and Python code. Uses custom authentication scheme for requests. Uses gpg under the hood. Is targeted at signing RPMs.
• robosignatory is tied to AWS S3 for both artifacts and signatures. Uses sigul underneath.
• dak shells out to gpg, seems to be aimed at software keys.
• code-signing aims at EFI code signing only using software keys. There is commented code that attempts to use Yubikeys.
• efi_signer works with software keys only and is aimed at just one specific area of signing.
• turnkey seems to be a “product” rather than something that can be easily deployed on premise. While it is possible that it can be adjusted for code-signing the primary use-case seems to be securing blockchain wallets.