As far the “how did you learn about Qubes” questions go, I think the conclusion on my side (as one of the authors of the survey) is simple: I really should have just included a “from Edward Snowden” option there, which would have saved our respondents some typing!
The survey covered many more questions than are described above, and they are all important for us as a team to learn more about you, our users, to know what to focus on and what to work on, what will work best and what may not be a great idea.
Don’t worry. If a question isn’t discussed above, we have still read it!
And again, thank you everyone for participating (and the many, many kind words you shared in your surveys).
The survey covered many more questions than are described above, and they are all important for us as a team to learn more about you, our users, to know what to focus on and what to work on, what will work best and what may not be a great idea.
Don’t worry. If a question isn’t discussed above, we have still read it!
And again, thank you everyone for participating (and the many, many kind words you shared in your surveys).
Qubes Canary 025
https://www.qubes-os.org/news/2020/12/12/canary-025/
We have published Qubes Canary 025. The text of this canary is
reproduced below.
Note: We have decided to make some minor formatting changes to the way
Qubes Canary and Qubes Security Bulletin (QSB) numbers are printed,
such as dropping the ‘#’ symbol and using hyphens instead of spaces.
This canary and its accompanying signatures will always be available in
the Qubes Security Pack (qubes-secpack).
View Qubes Canary 025 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/canaries/canary-025-2020.txt
Learn about the qubes-secpack, including how to obtain, verify, and
read it:
https://www.qubes-os.org/security/pack/
View all past canaries:
https://www.qubes-os.org/security/canaries/
---===[ Qubes Canary 025 ]===---
Statements
-----------
The Qubes core developers who have digitally signed this file [1]
state the following:
1. The date of issue of this canary is December 8, 2020.
2. There have been 62 Qubes Security Bulletins published so far.
3. The Qubes Master Signing Key fingerprint is:
427F 11FD 0FAA 4B08 0123 F01C DDFA 1A3E 3687 9494
4. No warrants have ever been served to us with regard to the Qubes OS
Project (e.g. to hand out the private signing keys or to introduce
backdoors).
5. We plan to publish the next of these canary statements in the first
two weeks of March 2020. Special note should be taken if no new canary
is published by that time or if the list of statements changes without
plausible explanation.
Special announcements
----------------------
None.
Disclaimers and notes
----------------------
We would like to remind you that Qubes OS has been designed under the
assumption that all relevant infrastructure is permanently
compromised. This means that we assume NO trust in any of the servers
or services which host or provide any Qubes-related data, in
particular, software updates, source code repositories, and Qubes ISO
downloads.
This canary scheme is not infallible. Although signing the declaration
makes it very difficult for a third party to produce arbitrary
declarations, it does not prevent them from using force or other
means, like blackmail or compromising the signers' laptops, to coerce
us to produce false declarations.
The news feeds quoted below (Proof of freshness) serves to demonstrate
that this canary could not have been created prior to the date stated.
It shows that a series of canaries was not created in advance.
This declaration is merely a best effort and is provided without any
guarantee or warranty. It is not legally binding in any way to
anybody. None of the signers should be ever held legally responsible
for any of the statements made here.
Proof of freshness
-------------------
Tue, 08 Dec 2020 16:46:42 +0000
Source: DER SPIEGEL - International (https://www.spiegel.de/international/index.rss)
Dangerous Accusations: German Tennis Star Alexander Zverev Faces Career Turning Point
Skiing in the Pandemic: Alpine Rivalries Flare amid Resort Closures
Biden's Goal of Saving the Iran Deal Just Got Harder - A Murder and an Ultimatum
Heiko Maas: Germany's Foreign Minister on the Future of Trans-Atlantic Relations
Generation Corona: The Pandemic Is Changing Our Children's Lives for the Worse
Source: NYT > World News (https://rss.nytimes.com/services/xml/rss/nyt/World.xml)
Covid-19 Live Updates: Britain Begins Vaccinating Citizens
U.K. Covid Vaccine: Side Effects, Safety, and Who Gets It First
U.S. Leaves Behind Afghan Bases — and a Legacy of Land Disputes
Covid Infections, and Blame, Rise Along Southeast Asian Borders
U.S. Imposes Sanctions on Chinese Officials Over Hong Kong Crackdown
Source: BBC News - World (https://feeds.bbci.co.uk/news/world/rss.xml)
Safety data on Pfizer jab released by US
Lloyd Austin: Biden picks ex-general as defence secretary
The man saving monkeys in the Colombian Amazon
https://www.qubes-os.org/news/2020/12/12/canary-025/
We have published Qubes Canary 025. The text of this canary is
reproduced below.
Note: We have decided to make some minor formatting changes to the way
Qubes Canary and Qubes Security Bulletin (QSB) numbers are printed,
such as dropping the ‘#’ symbol and using hyphens instead of spaces.
This canary and its accompanying signatures will always be available in
the Qubes Security Pack (qubes-secpack).
View Qubes Canary 025 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/canaries/canary-025-2020.txt
Learn about the qubes-secpack, including how to obtain, verify, and
read it:
https://www.qubes-os.org/security/pack/
View all past canaries:
https://www.qubes-os.org/security/canaries/
---===[ Qubes Canary 025 ]===---
Statements
-----------
The Qubes core developers who have digitally signed this file [1]
state the following:
1. The date of issue of this canary is December 8, 2020.
2. There have been 62 Qubes Security Bulletins published so far.
3. The Qubes Master Signing Key fingerprint is:
427F 11FD 0FAA 4B08 0123 F01C DDFA 1A3E 3687 9494
4. No warrants have ever been served to us with regard to the Qubes OS
Project (e.g. to hand out the private signing keys or to introduce
backdoors).
5. We plan to publish the next of these canary statements in the first
two weeks of March 2020. Special note should be taken if no new canary
is published by that time or if the list of statements changes without
plausible explanation.
Special announcements
----------------------
None.
Disclaimers and notes
----------------------
We would like to remind you that Qubes OS has been designed under the
assumption that all relevant infrastructure is permanently
compromised. This means that we assume NO trust in any of the servers
or services which host or provide any Qubes-related data, in
particular, software updates, source code repositories, and Qubes ISO
downloads.
This canary scheme is not infallible. Although signing the declaration
makes it very difficult for a third party to produce arbitrary
declarations, it does not prevent them from using force or other
means, like blackmail or compromising the signers' laptops, to coerce
us to produce false declarations.
The news feeds quoted below (Proof of freshness) serves to demonstrate
that this canary could not have been created prior to the date stated.
It shows that a series of canaries was not created in advance.
This declaration is merely a best effort and is provided without any
guarantee or warranty. It is not legally binding in any way to
anybody. None of the signers should be ever held legally responsible
for any of the statements made here.
Proof of freshness
-------------------
Tue, 08 Dec 2020 16:46:42 +0000
Source: DER SPIEGEL - International (https://www.spiegel.de/international/index.rss)
Dangerous Accusations: German Tennis Star Alexander Zverev Faces Career Turning Point
Skiing in the Pandemic: Alpine Rivalries Flare amid Resort Closures
Biden's Goal of Saving the Iran Deal Just Got Harder - A Murder and an Ultimatum
Heiko Maas: Germany's Foreign Minister on the Future of Trans-Atlantic Relations
Generation Corona: The Pandemic Is Changing Our Children's Lives for the Worse
Source: NYT > World News (https://rss.nytimes.com/services/xml/rss/nyt/World.xml)
Covid-19 Live Updates: Britain Begins Vaccinating Citizens
U.K. Covid Vaccine: Side Effects, Safety, and Who Gets It First
U.S. Leaves Behind Afghan Bases — and a Legacy of Land Disputes
Covid Infections, and Blame, Rise Along Southeast Asian Borders
U.S. Imposes Sanctions on Chinese Officials Over Hong Kong Crackdown
Source: BBC News - World (https://feeds.bbci.co.uk/news/world/rss.xml)
Safety data on Pfizer jab released by US
Lloyd Austin: Biden picks ex-general as defence secretary
The man saving monkeys in the Colombian Amazon
Charlie Hebdo attack: France seeks long jail terms in Paris trial
Christchurch massacre: Inquiry finds failures ahead of attack
Source: Blockchain.info
0000000000000000000c6550025327ca735099e0c621a9ad4599a49dab41f573
Footnotes
----------
[1] This file should be signed in two ways: (1) via detached PGP
signatures by each of the signers, distributed together with this
canary in the qubes-secpack.git repo, and (2) via digital signatures
on the corresponding qubes-secpack.git repo tags. [2]
[2] Don't just trust the contents of this file blindly! Verify the
digital signatures!
Christchurch massacre: Inquiry finds failures ahead of attack
Source: Blockchain.info
0000000000000000000c6550025327ca735099e0c621a9ad4599a49dab41f573
Footnotes
----------
[1] This file should be signed in two ways: (1) via detached PGP
signatures by each of the signers, distributed together with this
canary in the qubes-secpack.git repo, and (2) via digital signatures
on the corresponding qubes-secpack.git repo tags. [2]
[2] Don't just trust the contents of this file blindly! Verify the
digital signatures!
HVMI: Security Solutions Thrive on Friendly Hypervisors
https://xenproject.org/2020/12/14/hvmi-security-solutions-thrive-on-friendly-hypervisors/
This talk was given by Raul Tosa & Daniel Ticle, Bitdefender at the Xen Developer and Design Summit in July 2020. In July, Bitdefender open sourced Hypervisor Memory Introspection (HVMI)....
https://xenproject.org/2020/12/14/hvmi-security-solutions-thrive-on-friendly-hypervisors/
This talk was given by Raul Tosa & Daniel Ticle, Bitdefender at the Xen Developer and Design Summit in July 2020. In July, Bitdefender open sourced Hypervisor Memory Introspection (HVMI)....
QSB-063: Multiple Xen issues (XSA-115, XSA-325, XSA-350)
https://www.qubes-os.org/news/2020/12/16/qsb-063/
We have just published Qubes Security Bulletin (QSB) 063:
Stack corruption from XSA-346 change (XSA-355).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-063 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-063-2020.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View all XSAs mentioned in this QSB in the XSA Tracker:
https://www.qubes-os.org/security/xsa/
---===[ Qubes Security Bulletin 063 ]===---
2020-12-15
Multiple Xen issues (XSA-115, XSA-325, XSA-350)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Xen packages, version 4.8.5-28
- Linux kernel packages, versions 5.9.14-1, 5.4.83-1, 4.19.163-1
For Qubes 4.1:
- Xen packages, version 4.14.0-9
- Linux kernel packages, versions 5.9.14-1, 5.4.83-1, 4.19.163-1
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Xen binaries.
Summary
========
On 2020-12-15, the Xen Security Team published the following Xen
Security Advisories (XSAs):
XSA-115 [1] "xenstore watch notifications lacking permission checks"
| Neither xenstore implementation does any permissions checks when
| reporting a xenstore watch event.
|
| A guest administrator can watch the root xenstored node, which will
| cause notifications for every created, modified and deleted key.
|
| A guest administrator can also use the special watches, which will
| cause a notification every time a domain is created and destroyed.
|
| Data may include:
| - number, type and domids of other VMs
| - existence and domids of driver domains
| - numbers of virtual interfaces, block devices, vcpus
| - existence of virtual framebuffers and their backend style (eg,
| existence of VNC service)
| - Xen VM UUIDs for other domains
| - timing information about domain creation and device setup
| - some hints at the backend provisioning of VMs and their devices
|
| The watch events do not contain values stored in xenstore, only key
| names.
XSA-325 [2] "Xenstore: guests can disturb domain cleanup"
| Xenstored and guests communicate via a shared memory page using a
| specific protocol. When a guest violates this protocol, xenstored will
| drop the connection to that guest.
|
| Unfortunately this is done by just removing the guest from xenstored's
| internal management, resulting in the same actions as if the guest had
| been destroyed, including sending an @releaseDomain event.
|
| @releaseDomain events do not say guest has been removed. All watchers
| of this event must look at the states of all guests to find the guest
| which has been removed. When an @releaseDomain is generated due to
| domain xenstored protocol violation, As the guest is still running, so
| the watchers will not react.
|
| Later, when the guest is actually destroyed, xenstored will no longer
https://www.qubes-os.org/news/2020/12/16/qsb-063/
We have just published Qubes Security Bulletin (QSB) 063:
Stack corruption from XSA-346 change (XSA-355).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-063 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-063-2020.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View all XSAs mentioned in this QSB in the XSA Tracker:
https://www.qubes-os.org/security/xsa/
---===[ Qubes Security Bulletin 063 ]===---
2020-12-15
Multiple Xen issues (XSA-115, XSA-325, XSA-350)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Xen packages, version 4.8.5-28
- Linux kernel packages, versions 5.9.14-1, 5.4.83-1, 4.19.163-1
For Qubes 4.1:
- Xen packages, version 4.14.0-9
- Linux kernel packages, versions 5.9.14-1, 5.4.83-1, 4.19.163-1
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Xen binaries.
Summary
========
On 2020-12-15, the Xen Security Team published the following Xen
Security Advisories (XSAs):
XSA-115 [1] "xenstore watch notifications lacking permission checks"
| Neither xenstore implementation does any permissions checks when
| reporting a xenstore watch event.
|
| A guest administrator can watch the root xenstored node, which will
| cause notifications for every created, modified and deleted key.
|
| A guest administrator can also use the special watches, which will
| cause a notification every time a domain is created and destroyed.
|
| Data may include:
| - number, type and domids of other VMs
| - existence and domids of driver domains
| - numbers of virtual interfaces, block devices, vcpus
| - existence of virtual framebuffers and their backend style (eg,
| existence of VNC service)
| - Xen VM UUIDs for other domains
| - timing information about domain creation and device setup
| - some hints at the backend provisioning of VMs and their devices
|
| The watch events do not contain values stored in xenstore, only key
| names.
XSA-325 [2] "Xenstore: guests can disturb domain cleanup"
| Xenstored and guests communicate via a shared memory page using a
| specific protocol. When a guest violates this protocol, xenstored will
| drop the connection to that guest.
|
| Unfortunately this is done by just removing the guest from xenstored's
| internal management, resulting in the same actions as if the guest had
| been destroyed, including sending an @releaseDomain event.
|
| @releaseDomain events do not say guest has been removed. All watchers
| of this event must look at the states of all guests to find the guest
| which has been removed. When an @releaseDomain is generated due to
| domain xenstored protocol violation, As the guest is still running, so
| the watchers will not react.
|
| Later, when the guest is actually destroyed, xenstored will no longer
| have it stored in its internal data base, so no further @releaseDomain
| event will be sent. This can lead to a zombie domain; memory mappings
| of that guest's memory will not be removed, due to the missing
| event. This zombie domain will be cleaned up only after another domain
| is destroyed, as that will trigger another @releaseDomain event.
|
| If the device model of the guest which violated the Xenstore protocol
| is running in a stub-domain, a use-after-free case could happen in
| xenstored, after having removed the guest from its internal data base,
| possibly resulting in a crash of xenstored.
XSA-350 [3] "Use after free triggered by block frontend in Linux blkback"
| The Linux kernel PV block backend expects the kernel thread handler
| to reset ring->xenblkd to NULL when stopped. However, the handler may
| not have time to run if the frontend quickly toggle between the states
| connect and disconnect.
|
| As a consequence, the block backend may re-use a pointer after it was
| freed.
Impact
=======
XSA-115, as described by Xen Security Team:
| A guest administrator can observe non-sensitive domain and device
| lifecycle events relating to other guests. This information allows
| some insight into overall system configuration (including number of
| general nature of other guests), and configuration of other guests
| (including number and general nature of other guests' devices). This
| information might be commercially interesting or might make other
| attacks easier.
|
| There is not believed to be exposure of sensitive data. Specifically,
| there is no exposure of: VNC passwords; port numbers; pathnames in host
| and guest filesystems; cryptopgraphic keys; or within-guest data.
XSA-325:
This issue allows a compromised domain to delay resource cleanup,
including disposing DisposableVMs. The delay will be at most until
another domain shutdown. If an HVM domain (like sys-net or sys-usb)
triggers it, it may cause a use-after-free issue in the xenstored
daemon. It may cause the daemon to crash (preventing most further
operations in the system). Beyond DoS, it is unlikely that this
vulnerability could be exploited to compromise the system, but we
cannot completely rule out the possibility.
XSA-350, as described by Xen Security Team:
| A misbehaving guest can trigger a dom0 crash by continuously
| connecting / disconnecting a block frontend. Privileged escalation and
| information leak cannot be ruled out.
Credits
========
See the original Xen Security Advisories.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-115.html
[2] https://xenbits.xen.org/xsa/advisory-325.html
[3] https://xenbits.xen.org/xsa/advisory-350.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
| event will be sent. This can lead to a zombie domain; memory mappings
| of that guest's memory will not be removed, due to the missing
| event. This zombie domain will be cleaned up only after another domain
| is destroyed, as that will trigger another @releaseDomain event.
|
| If the device model of the guest which violated the Xenstore protocol
| is running in a stub-domain, a use-after-free case could happen in
| xenstored, after having removed the guest from its internal data base,
| possibly resulting in a crash of xenstored.
XSA-350 [3] "Use after free triggered by block frontend in Linux blkback"
| The Linux kernel PV block backend expects the kernel thread handler
| to reset ring->xenblkd to NULL when stopped. However, the handler may
| not have time to run if the frontend quickly toggle between the states
| connect and disconnect.
|
| As a consequence, the block backend may re-use a pointer after it was
| freed.
Impact
=======
XSA-115, as described by Xen Security Team:
| A guest administrator can observe non-sensitive domain and device
| lifecycle events relating to other guests. This information allows
| some insight into overall system configuration (including number of
| general nature of other guests), and configuration of other guests
| (including number and general nature of other guests' devices). This
| information might be commercially interesting or might make other
| attacks easier.
|
| There is not believed to be exposure of sensitive data. Specifically,
| there is no exposure of: VNC passwords; port numbers; pathnames in host
| and guest filesystems; cryptopgraphic keys; or within-guest data.
XSA-325:
This issue allows a compromised domain to delay resource cleanup,
including disposing DisposableVMs. The delay will be at most until
another domain shutdown. If an HVM domain (like sys-net or sys-usb)
triggers it, it may cause a use-after-free issue in the xenstored
daemon. It may cause the daemon to crash (preventing most further
operations in the system). Beyond DoS, it is unlikely that this
vulnerability could be exploited to compromise the system, but we
cannot completely rule out the possibility.
XSA-350, as described by Xen Security Team:
| A misbehaving guest can trigger a dom0 crash by continuously
| connecting / disconnecting a block frontend. Privileged escalation and
| information leak cannot be ruled out.
Credits
========
See the original Xen Security Advisories.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-115.html
[2] https://xenbits.xen.org/xsa/advisory-325.html
[3] https://xenbits.xen.org/xsa/advisory-350.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
Cut your Cloud Computing Costs by Half with Unikraft
https://xenproject.org/2020/12/18/cut-your-cloud-computing-costs-by-half-with-unikraft/
This post originally appeared on Linux.com A novel modular unikernel allows for extreme tailoring of your operating system to your application’s needs. A proof of concept, built on Unikraft, a...
https://xenproject.org/2020/12/18/cut-your-cloud-computing-costs-by-half-with-unikraft/
This post originally appeared on Linux.com A novel modular unikernel allows for extreme tailoring of your operating system to your application’s needs. A proof of concept, built on Unikraft, a...
XSAs released on 2021-01-21
https://www.qubes-os.org/news/2021/01/22/xsas-released-on-2021-01-21/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-01-21.
The security of Qubes OS is not affected by these XSAs.
Therefore, user action is not required.
XSAs that affect the security of Qubes OS (user action required)
None.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
XSA-360 (DoS only)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
https://www.qubes-os.org/news/2021/01/22/xsas-released-on-2021-01-21/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-01-21.
The security of Qubes OS is not affected by these XSAs.
Therefore, user action is not required.
XSAs that affect the security of Qubes OS (user action required)
None.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
XSA-360 (DoS only)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
The Xen Project at FOSDEM: Unikraft and XCP-ng
https://xenproject.org/2021/02/04/the-xen-project-at-fosdem-unikraft-and-xcp-ng/
It’s that time of year when open source community gathers for FOSDEM! FOSDEM is a free event for software developers to meet, share ideas, and collaborate. This year’s event will...
https://xenproject.org/2021/02/04/the-xen-project-at-fosdem-unikraft-and-xcp-ng/
It’s that time of year when open source community gathers for FOSDEM! FOSDEM is a free event for software developers to meet, share ideas, and collaborate. This year’s event will...
2021: Xen Project, virtualization and beyond
https://xenproject.org/2021/02/10/2021-xen-project-virtualization-and-beyond/
This post originally appeared on VM Blog. By George Dunlap, Advisory Board Chair for the Xen Project The Xen Project has been around for the better part of two decades. As...
https://xenproject.org/2021/02/10/2021-xen-project-virtualization-and-beyond/
This post originally appeared on VM Blog. By George Dunlap, Advisory Board Chair for the Xen Project The Xen Project has been around for the better part of two decades. As...
QSB-064: Linux: error handling issues in blkback's grant mapping (XSA-365)
https://www.qubes-os.org/news/2021/02/17/qsb-064/
We have just published Qubes Security Bulletin (QSB) 064:
Linux: error handling issues in blkback’s grant mapping (XSA-365).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-064 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-064-2021.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View XSA-365 in the XSA Tracker:
https://www.qubes-os.org/security/xsa/#365
---===[ Qubes Security Bulletin 064 ]===---
2021-02-16
Linux: error handling issues in blkback's grant mapping (XSA-365)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Linux kernel packages, versions 5.10.16-1, 5.4.98-1, 4.19.176-1
For Qubes 4.1:
- Linux kernel packages, versions 5.10.16-1, 5.4.98-1
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Linux kernel binaries.
Summary
========
On 2021-02-16, the Xen Security Team published the following Xen
Security Advisory (XSA):
XSA-365 [1] "Linux: error handling issues in blkback's grant mapping"
| To service requests, the driver maps grant references provided by the
| frontend. In this process, errors may be encountered. In one case an
| error encountered earlier might be discarded by later processing,
| resulting in the caller assuming successful mapping, and hence
| subsequent operations trying to access space that wasn't mapped. In
| another case internal state would be insufficiently updated, preventing
| safe recovery from the error.
Impact
=======
XSA-365, as described by Xen Security Team:
| A malicious or buggy frontend driver may be able to crash the
| corresponding backend driver, potentially affecting the entire domain
| running the backend driver. In configurations without driver domains
| or similar disaggregation, that is a host-wide denial of sevice.
|
| Privilege escalation and information leaks cannot be ruled out.
Credits
========
See the original Xen Security Advisories.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-365.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
https://www.qubes-os.org/news/2021/02/17/qsb-064/
We have just published Qubes Security Bulletin (QSB) 064:
Linux: error handling issues in blkback’s grant mapping (XSA-365).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-064 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-064-2021.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View XSA-365 in the XSA Tracker:
https://www.qubes-os.org/security/xsa/#365
---===[ Qubes Security Bulletin 064 ]===---
2021-02-16
Linux: error handling issues in blkback's grant mapping (XSA-365)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Linux kernel packages, versions 5.10.16-1, 5.4.98-1, 4.19.176-1
For Qubes 4.1:
- Linux kernel packages, versions 5.10.16-1, 5.4.98-1
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Linux kernel binaries.
Summary
========
On 2021-02-16, the Xen Security Team published the following Xen
Security Advisory (XSA):
XSA-365 [1] "Linux: error handling issues in blkback's grant mapping"
| To service requests, the driver maps grant references provided by the
| frontend. In this process, errors may be encountered. In one case an
| error encountered earlier might be discarded by later processing,
| resulting in the caller assuming successful mapping, and hence
| subsequent operations trying to access space that wasn't mapped. In
| another case internal state would be insufficiently updated, preventing
| safe recovery from the error.
Impact
=======
XSA-365, as described by Xen Security Team:
| A malicious or buggy frontend driver may be able to crash the
| corresponding backend driver, potentially affecting the entire domain
| running the backend driver. In configurations without driver domains
| or similar disaggregation, that is a host-wide denial of sevice.
|
| Privilege escalation and information leaks cannot be ruled out.
Credits
========
See the original Xen Security Advisories.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-365.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
XSAs released on 2021-02-16
https://www.qubes-os.org/news/2021/02/17/xsas-released-on-2021-02-16/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-02-16.
The security of Qubes OS is affected by one or more of these XSAs.
Therefore, user action is required.
XSAs that affect the security of Qubes OS (user action required)
The following XSAs do affect the security of Qubes OS:
XSA-365
Please see QSB-064 (https://www.qubes-os.org/news/2021/02/17/qsb-064/) for the actions users must take in order to protect themselves, as well as further details about these XSAs.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
XSA-361 (DoS-only)
XSA-362 (DoS-only)
XSA-363 (DoS-only)
XSA-364 (ARM-only)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
https://www.qubes-os.org/news/2021/02/17/xsas-released-on-2021-02-16/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-02-16.
The security of Qubes OS is affected by one or more of these XSAs.
Therefore, user action is required.
XSAs that affect the security of Qubes OS (user action required)
The following XSAs do affect the security of Qubes OS:
XSA-365
Please see QSB-064 (https://www.qubes-os.org/news/2021/02/17/qsb-064/) for the actions users must take in order to protect themselves, as well as further details about these XSAs.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
XSA-361 (DoS-only)
XSA-362 (DoS-only)
XSA-363 (DoS-only)
XSA-364 (ARM-only)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
QSB-065: Missed flush in XSA-321 backport (XSA-366)
https://www.qubes-os.org/news/2021/02/19/qsb-065/
We have just published Qubes Security Bulletin (QSB) 065:
Missed flush in XSA-321 backport (XSA-366).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-065 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-065-2021.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View XSA-366 in the XSA Tracker:
https://www.qubes-os.org/security/xsa/#366
---===[ Qubes Security Bulletin 065 ]===---
2021-02-18
Missed flush in XSA-321 backport (XSA-366)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Xen packages, versions 4.8.5-30
For Qubes 4.1: not affected
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Xen binaries.
Summary
========
On 2021-02-18, the Xen Security Team published the following Xen
Security Advisory (XSA):
XSA-366 [1] "missed flush in XSA-321 backport"
| An oversight was made when backporting XSA-320, leading entries in the
| IOMMU not being properly updated under certain circumstances.
Impact
=======
XSA-366, as described by the Xen Security Team:
| A malicious guest may be able to retain read/write DMA access to
| frames returned to Xen's free pool, and later reused for another
| purpose. Host crashes (leading to a Denial of Service) and privilege
| escalation cannot be ruled out.
Credits
========
See the original Xen Security Advisory.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-366.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
https://www.qubes-os.org/news/2021/02/19/qsb-065/
We have just published Qubes Security Bulletin (QSB) 065:
Missed flush in XSA-321 backport (XSA-366).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-065 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-065-2021.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View XSA-366 in the XSA Tracker:
https://www.qubes-os.org/security/xsa/#366
---===[ Qubes Security Bulletin 065 ]===---
2021-02-18
Missed flush in XSA-321 backport (XSA-366)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Xen packages, versions 4.8.5-30
For Qubes 4.1: not affected
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Xen binaries.
Summary
========
On 2021-02-18, the Xen Security Team published the following Xen
Security Advisory (XSA):
XSA-366 [1] "missed flush in XSA-321 backport"
| An oversight was made when backporting XSA-320, leading entries in the
| IOMMU not being properly updated under certain circumstances.
Impact
=======
XSA-366, as described by the Xen Security Team:
| A malicious guest may be able to retain read/write DMA access to
| frames returned to Xen's free pool, and later reused for another
| purpose. Host crashes (leading to a Denial of Service) and privilege
| escalation cannot be ruled out.
Credits
========
See the original Xen Security Advisory.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-366.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
XSAs released on 2021-02-18
https://www.qubes-os.org/news/2021/02/19/xsas-released-on-2021-02-18/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-02-18.
The security of Qubes OS is affected by one or more of these XSAs.
Therefore, user action is required.
XSAs that affect the security of Qubes OS (user action required)
The following XSAs do affect the security of Qubes OS:
XSA-366
Please see QSB-065 (https://www.qubes-os.org/news/2021/02/19/qsb-065/) for the actions users must take in order to protect themselves, as well as further details about these XSAs.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
(None)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
https://www.qubes-os.org/news/2021/02/19/xsas-released-on-2021-02-18/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-02-18.
The security of Qubes OS is affected by one or more of these XSAs.
Therefore, user action is required.
XSAs that affect the security of Qubes OS (user action required)
The following XSAs do affect the security of Qubes OS:
XSA-366
Please see QSB-065 (https://www.qubes-os.org/news/2021/02/19/qsb-065/) for the actions users must take in order to protect themselves, as well as further details about these XSAs.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
(None)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
Fedora 33 TemplateVMs available
https://www.qubes-os.org/news/2021/02/25/fedora-33-templates-available/
New Fedora 33 TemplateVMs are now available for both Qubes 4.0 and 4.1.
Important: If you wish to use the Qubes Update widget to update a
Fedora 33 template, you must first switch (https://www.qubes-os.org/doc/templates/#switching) the
default-mgmt-dvm qube to a Fedora 33 template. (Alternatively, you
can create a separate management DisposableVM Template based on a
Fedora 33 template for the purpose of updating Fedora 33 templates.)
This does not affect updating internally using dnf.
Instructions are available for upgrading Fedora TemplateVMs (https://www.qubes-os.org/doc/template/fedora/upgrade/). We also
provide fresh Fedora 33 TemplateVM packages through the official Qubes
repositories, which you can get with the following commands (in dom0).
Standard (https://www.qubes-os.org/doc/templates/fedora/) Fedora 33 TemplateVM:
$ sudo qubes-dom0-update qubes-template-fedora-33
Minimal (https://www.qubes-os.org/doc/templates/minimal/) Fedora 33 TemplateVM:
$ sudo qubes-dom0-update qubes-template-fedora-33-minimal
After installing or upgrading a TemplateVM, please remember to update (https://www.qubes-os.org/doc/software-update-domu/)
(see important note above) and switch all qubes that were using the
old template to use the new one (https://www.qubes-os.org/doc/templates/#switching).
https://www.qubes-os.org/news/2021/02/25/fedora-33-templates-available/
New Fedora 33 TemplateVMs are now available for both Qubes 4.0 and 4.1.
Important: If you wish to use the Qubes Update widget to update a
Fedora 33 template, you must first switch (https://www.qubes-os.org/doc/templates/#switching) the
default-mgmt-dvm qube to a Fedora 33 template. (Alternatively, you
can create a separate management DisposableVM Template based on a
Fedora 33 template for the purpose of updating Fedora 33 templates.)
This does not affect updating internally using dnf.
Instructions are available for upgrading Fedora TemplateVMs (https://www.qubes-os.org/doc/template/fedora/upgrade/). We also
provide fresh Fedora 33 TemplateVM packages through the official Qubes
repositories, which you can get with the following commands (in dom0).
Standard (https://www.qubes-os.org/doc/templates/fedora/) Fedora 33 TemplateVM:
$ sudo qubes-dom0-update qubes-template-fedora-33
Minimal (https://www.qubes-os.org/doc/templates/minimal/) Fedora 33 TemplateVM:
$ sudo qubes-dom0-update qubes-template-fedora-33-minimal
After installing or upgrading a TemplateVM, please remember to update (https://www.qubes-os.org/doc/software-update-domu/)
(see important note above) and switch all qubes that were using the
old template to use the new one (https://www.qubes-os.org/doc/templates/#switching).
Improvements in testing and building: GitLab CI and reproducible builds
https://www.qubes-os.org/news/2021/02/28/improvements-in-testing-and-building/
Over the last couple of months, we have made some significant changes to two important parts of the Qubes development process: testing and building.
What are continuous integration (CI) and reproducible builds?
Automated testing is a major part of the software development process. It spares developers many, many hours of manual testing that would still miss some bugs and other problems. In Qubes development, we’re using an approach called “continuous integration” (CI), in which local changes made by the developers are frequently merged and tested remotely, using dedicated automated testing solutions. This is very important both for maintaining consistent code quality (all changes are tested) and for making development easier for the developers. Testing Qubes is not easy. Since Qubes is an entire operating system, doing the testing on the same system in which you’re developing is a bit like building a rocket landing system en route to Mars — not impossible, but very stressful.
The second area of improvement is the build process. The term “reproducible builds (https://reproducible-builds.org/)” refers to a process in which the same source code always compiles into exactly the same binary (for example, a package used to install a program via a package manager like dnf or apt). Why is this difficult to achieve? After all, computers are not random. Shouldn’t builds be reproducible by default, without requiring special effort to make them deterministic? Unfortunately, it’s not that simple. There are thousands of variables influencing the way binaries are built, ranging from the time of day to the availability of remote servers and locale settings.
Ensuring that binaries are built the same way every time is surprisingly difficult. However, the effort is worth the security benefits. To understand these benefits, imagine that an attacker wishes to feed unsuspecting users a compromised package. The attacker knows that the source code is public, so any malicious code he inserts into it would be highly exposed and at risk of detection. On the other hand, he reasons, compromising the build infrastructure would allow him to surreptitiously insert malicious changes that would make it into the resultant package. Since the source code remains untouched, his malicious changes are less likely to be detected. This is where the value of reproducible builds comes in. If the build process is reproducible, then we will immediately notice that building a package from the untouched source code results in a package that is different from the compromised one. This would be a major red flag that would prompt an immediate security investigation.
GitLab-CI migration
As many of you are aware, we migrated from Travis-CI to GitLab-CI late last year. While the direct reason (https://www.mail-archive.com/qubes-devel@googlegroups.com/msg04698.html) was a change in the Travis-CI terms of service, GitLab-CI gives us many additional benefits. Just to name a few:
A modern execution environment with native Docker support: We can use whatever base environment we like. We are no longer constrained to specific (not so fresh) Ubuntu versions.
Much more flexible job definitions, including dependencies among them: We use this to split jobs into smaller pieces that can run in parallel and reduce duplication among them.
Out-of-the-box support for caching and artifacts: Another feature allowing for a great speed-up of our tests. A specific build environment can be stored with a pre-populated cache, for example avoiding the need to create a chroot environment each time.
Higher time limits and the ability to connect our own workers: This allows us to automatically test bigger components like the Linux kernel (which previously didn’t fit into Travis-CI’s hard time limit).
https://www.qubes-os.org/news/2021/02/28/improvements-in-testing-and-building/
Over the last couple of months, we have made some significant changes to two important parts of the Qubes development process: testing and building.
What are continuous integration (CI) and reproducible builds?
Automated testing is a major part of the software development process. It spares developers many, many hours of manual testing that would still miss some bugs and other problems. In Qubes development, we’re using an approach called “continuous integration” (CI), in which local changes made by the developers are frequently merged and tested remotely, using dedicated automated testing solutions. This is very important both for maintaining consistent code quality (all changes are tested) and for making development easier for the developers. Testing Qubes is not easy. Since Qubes is an entire operating system, doing the testing on the same system in which you’re developing is a bit like building a rocket landing system en route to Mars — not impossible, but very stressful.
The second area of improvement is the build process. The term “reproducible builds (https://reproducible-builds.org/)” refers to a process in which the same source code always compiles into exactly the same binary (for example, a package used to install a program via a package manager like dnf or apt). Why is this difficult to achieve? After all, computers are not random. Shouldn’t builds be reproducible by default, without requiring special effort to make them deterministic? Unfortunately, it’s not that simple. There are thousands of variables influencing the way binaries are built, ranging from the time of day to the availability of remote servers and locale settings.
Ensuring that binaries are built the same way every time is surprisingly difficult. However, the effort is worth the security benefits. To understand these benefits, imagine that an attacker wishes to feed unsuspecting users a compromised package. The attacker knows that the source code is public, so any malicious code he inserts into it would be highly exposed and at risk of detection. On the other hand, he reasons, compromising the build infrastructure would allow him to surreptitiously insert malicious changes that would make it into the resultant package. Since the source code remains untouched, his malicious changes are less likely to be detected. This is where the value of reproducible builds comes in. If the build process is reproducible, then we will immediately notice that building a package from the untouched source code results in a package that is different from the compromised one. This would be a major red flag that would prompt an immediate security investigation.
GitLab-CI migration
As many of you are aware, we migrated from Travis-CI to GitLab-CI late last year. While the direct reason (https://www.mail-archive.com/qubes-devel@googlegroups.com/msg04698.html) was a change in the Travis-CI terms of service, GitLab-CI gives us many additional benefits. Just to name a few:
A modern execution environment with native Docker support: We can use whatever base environment we like. We are no longer constrained to specific (not so fresh) Ubuntu versions.
Much more flexible job definitions, including dependencies among them: We use this to split jobs into smaller pieces that can run in parallel and reduce duplication among them.
Out-of-the-box support for caching and artifacts: Another feature allowing for a great speed-up of our tests. A specific build environment can be stored with a pre-populated cache, for example avoiding the need to create a chroot environment each time.
Higher time limits and the ability to connect our own workers: This allows us to automatically test bigger components like the Linux kernel (which previously didn’t fit into Travis-CI’s hard time limit).
We still host the actual code on GitHub. We use GitLab only for CI. This mode of operation is supported natively by GitLab, but this support is quite limited. Most importantly, it does not support (https://gitlab.com/gitlab-org/gitlab/-/issues/5667) testing pull requests made from repository forks, which is the vast majority of our pull requests (if not all of them). For this reason, Frédéric ended up creating our own integration (https://github.com/fepitre/qubes-g2g-continuous-integration), which takes care of uploading the code to GitLab, scheduling CI jobs, and reporting the results back to GitHub. In fact, we are not the only project that has taken this route. Several other similar projects, such as check_pr, coq bot, and labhub, have also done so.
In the process, our CI has gained some nice extra features:
Control via special comments in pull requests:
PipelineRetry — restarts a CI job
PipelineRefresh — refreshes job status in GitHub (should normally not be needed)
testdeploy — in a website-related pull request, takes the website built in GitLab-CI and publishes it under a temporary address (works only after the GitLab-CI build completes and only for selected users)
More tests, specifically for reproducible builds of packages
We also have a nice summary page (https://qubesos.gitlab.io/qubes-g2g-report/) with all the recent jobs states, which is updated daily.
Reproducible builds
Our current short-term goal for reproducible builds in Qubes OS is to integrate what is realistically possible in Debian. Specifically, we aim to get our Debian templates to the state in which packages can be installed only when configured rebuilders confirm they really came from the source code we publish. (A “rebuilder” is a program that takes a binary and its purported source code as inputs, along with any applicable metadata, and attempts to build an identical binary from the source code. The goal is to check whether the binary was really compiled from its claimed source code.) For this task, we focus specifically on Debian Bullseye (testing version), which includes most of the reproducibility work done by the Debian community.
To start, we had to bring our packages to the state in which they could be built reproducibly. GitLab-CI massively helped here with identifying issues. One of the CI jobs we added is running the reprotest tool on our packages. This tool builds the package twice in two different build environments and compares the results. If there are any differences, the job automatically runs the diffoscope tool, which provides a detailed, in-depth report.
Now, all our packages for Debian Bullseye are reproducible, and we are actively monitoring it with our CI. This is designated by the “reproducible” badge on the CI summary page (https://qubesos.gitlab.io/qubes-g2g-report/).
The next step on the journey is to enable an arbitrary user or organization to challenge the official packages — to take the sources, together with the buildinfo file describing how it was built and to verify that they result in the same binary package. Theoretically, there is a debrebuild.pl noscript in Debian’s devnoscripts package that does exactly this. We started by adding missing parts to it, but it turned out there were several limitations making it unsuitable for our use case. The primary limitation is support for only one source repository. (We need two: original Debian and Qubes.) Since debrebuild.pl, as the name suggests, is written in Perl, it isn’t very friendly for contributing to. For these reasons, we decided to write our own similar tool in Python. While this was a bit of work, the end result (https://github.com/fepitre/debrebuild) is very nice: about 900 lines of code, with proper structure using classes. We are in the process of upstreaming this code back to Debian, potentially to replace the original debrebuild.pl tool.
In the process, our CI has gained some nice extra features:
Control via special comments in pull requests:
PipelineRetry — restarts a CI job
PipelineRefresh — refreshes job status in GitHub (should normally not be needed)
testdeploy — in a website-related pull request, takes the website built in GitLab-CI and publishes it under a temporary address (works only after the GitLab-CI build completes and only for selected users)
More tests, specifically for reproducible builds of packages
We also have a nice summary page (https://qubesos.gitlab.io/qubes-g2g-report/) with all the recent jobs states, which is updated daily.
Reproducible builds
Our current short-term goal for reproducible builds in Qubes OS is to integrate what is realistically possible in Debian. Specifically, we aim to get our Debian templates to the state in which packages can be installed only when configured rebuilders confirm they really came from the source code we publish. (A “rebuilder” is a program that takes a binary and its purported source code as inputs, along with any applicable metadata, and attempts to build an identical binary from the source code. The goal is to check whether the binary was really compiled from its claimed source code.) For this task, we focus specifically on Debian Bullseye (testing version), which includes most of the reproducibility work done by the Debian community.
To start, we had to bring our packages to the state in which they could be built reproducibly. GitLab-CI massively helped here with identifying issues. One of the CI jobs we added is running the reprotest tool on our packages. This tool builds the package twice in two different build environments and compares the results. If there are any differences, the job automatically runs the diffoscope tool, which provides a detailed, in-depth report.
Now, all our packages for Debian Bullseye are reproducible, and we are actively monitoring it with our CI. This is designated by the “reproducible” badge on the CI summary page (https://qubesos.gitlab.io/qubes-g2g-report/).
The next step on the journey is to enable an arbitrary user or organization to challenge the official packages — to take the sources, together with the buildinfo file describing how it was built and to verify that they result in the same binary package. Theoretically, there is a debrebuild.pl noscript in Debian’s devnoscripts package that does exactly this. We started by adding missing parts to it, but it turned out there were several limitations making it unsuitable for our use case. The primary limitation is support for only one source repository. (We need two: original Debian and Qubes.) Since debrebuild.pl, as the name suggests, is written in Perl, it isn’t very friendly for contributing to. For these reasons, we decided to write our own similar tool in Python. While this was a bit of work, the end result (https://github.com/fepitre/debrebuild) is very nice: about 900 lines of code, with proper structure using classes. We are in the process of upstreaming this code back to Debian, potentially to replace the original debrebuild.pl tool.
The next step is to configure apt-transport-in-toto to actually use these proofs, but this step requires the user to decide who to trust. The configuration process is currently described in the README of rebuilder orchestrator (https://github.com/fepitre/package-rebuilder/). It includes steps to use both Frédéric’s rebuilder as well as to create a custom configuration. When enabled, every installed package is checked to ensure that a rebuilder can confirm that the package was really compiled from its purported source code. For example, installing the tzdata package looks like this:
root@debian-11:~# apt-get install tzdata
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
tzdata
1 upgraded, 0 newly installed, 0 to remove and 842 not upgraded.
Need to get 284 kB of archives.
After this operation, 4,096 B of additional disk space will be used.
Get:1 intoto://deb.debian.org/debian bullseye/main amd64 tzdata all 2021a-1 [284 kB]
(...)
In-toto verification for '/var/cache/apt/archives/partial/tzdata_2021a-1_all.deb' passed! :)
Fetched 284 kB in 15s (19.3 kB/s)
Reading changelogs... Done
(...)
Reproducible RPMs
While we started with Debian, our goal is to have all of our packages reproducibly built and independently verified. Reproducible builds support in the RPM ecosystem is mainly driven by OpenSUSE, but Fedora and other RPM-based distributions benefit from this too. In some cases, this means that Fedora just needs some configuration to be adjusted in order to achieve a reproducible build, while in other cases this requires developing missing tools.
First of all, there is a set of tools that is very helpful while working on reproducible builds, specifically:
diffoscope — compares files in depth; an indispensable tool when working with packages and other compound file formats
reprotest — builds a package twice and compares the results
disorderfs — intentionally disturbs the way in which the filesystem is visible to applications, especially the order in which files are listed
While diffoscope was already available in Fedora, the other two weren’t. Frédéric started by adding RPM support to reprotest and then packaging both disorderfs and reprotest for Fedora. Thanks to these efforts, all three tools are now available in the default Fedora repositories.
Next, similarly to Debian, we developed a tool that takes an official package and tries to reproduce it. We call it rpmreproduce. While working on it, we noticed that there is no single official format for an RPM package build environment denoscription, something that has existed in Debian for a long time as the buildinfo format. After discussing it a bit with the community, we submitted a buildinfo proposal and implementation to the upstream RPM project. At this stage, the only thing left to do is to add it to the rebuilder orchestrator mentioned earlier.
Having a package rebuilder running for RPM packages is one thing. Actually using it to verify packages during installation is another matter. In Debian, we used an existing APT plugin (apt-transport-in-toto), but for Fedora no such thing existed. To fill this gap, we have developed dnf-plugin-in-toto which works on a very similar basis. The DNF plugin architecture is a bit different from that of APT. It allows DNF to request a rebuild confirmation (based on the package hash from the repository metadata) before actually downloading package. There is still some work to do on this plugin, but the basic functionality is there already. Here’s an example:
# dnf install --enablerepo=qubes*testing qubes-u2f
(...)
Prepare in-toto verification for 'qubes-u2f-1.2.8-1.fc33.noarch'
Create verification directory '/tmp/tmpfei9swbm'
Request in-toto metadata from 1 rebuilder(s) (DNF global_info)
Request in-toto metadata from https://mirror.notset.fr/qubes/rebuild/yum/r4.1/vm/sources/qubes-u2f/1.2.8-1.fc33/metadata
root@debian-11:~# apt-get install tzdata
Reading package lists... Done
Building dependency tree... Done
Reading state information... Done
The following packages will be upgraded:
tzdata
1 upgraded, 0 newly installed, 0 to remove and 842 not upgraded.
Need to get 284 kB of archives.
After this operation, 4,096 B of additional disk space will be used.
Get:1 intoto://deb.debian.org/debian bullseye/main amd64 tzdata all 2021a-1 [284 kB]
(...)
In-toto verification for '/var/cache/apt/archives/partial/tzdata_2021a-1_all.deb' passed! :)
Fetched 284 kB in 15s (19.3 kB/s)
Reading changelogs... Done
(...)
Reproducible RPMs
While we started with Debian, our goal is to have all of our packages reproducibly built and independently verified. Reproducible builds support in the RPM ecosystem is mainly driven by OpenSUSE, but Fedora and other RPM-based distributions benefit from this too. In some cases, this means that Fedora just needs some configuration to be adjusted in order to achieve a reproducible build, while in other cases this requires developing missing tools.
First of all, there is a set of tools that is very helpful while working on reproducible builds, specifically:
diffoscope — compares files in depth; an indispensable tool when working with packages and other compound file formats
reprotest — builds a package twice and compares the results
disorderfs — intentionally disturbs the way in which the filesystem is visible to applications, especially the order in which files are listed
While diffoscope was already available in Fedora, the other two weren’t. Frédéric started by adding RPM support to reprotest and then packaging both disorderfs and reprotest for Fedora. Thanks to these efforts, all three tools are now available in the default Fedora repositories.
Next, similarly to Debian, we developed a tool that takes an official package and tries to reproduce it. We call it rpmreproduce. While working on it, we noticed that there is no single official format for an RPM package build environment denoscription, something that has existed in Debian for a long time as the buildinfo format. After discussing it a bit with the community, we submitted a buildinfo proposal and implementation to the upstream RPM project. At this stage, the only thing left to do is to add it to the rebuilder orchestrator mentioned earlier.
Having a package rebuilder running for RPM packages is one thing. Actually using it to verify packages during installation is another matter. In Debian, we used an existing APT plugin (apt-transport-in-toto), but for Fedora no such thing existed. To fill this gap, we have developed dnf-plugin-in-toto which works on a very similar basis. The DNF plugin architecture is a bit different from that of APT. It allows DNF to request a rebuild confirmation (based on the package hash from the repository metadata) before actually downloading package. There is still some work to do on this plugin, but the basic functionality is there already. Here’s an example:
# dnf install --enablerepo=qubes*testing qubes-u2f
(...)
Prepare in-toto verification for 'qubes-u2f-1.2.8-1.fc33.noarch'
Create verification directory '/tmp/tmpfei9swbm'
Request in-toto metadata from 1 rebuilder(s) (DNF global_info)
Request in-toto metadata from https://mirror.notset.fr/qubes/rebuild/yum/r4.1/vm/sources/qubes-u2f/1.2.8-1.fc33/metadata
Successfully downloaded in-toto metadata 'rebuild.8deb0bef.link' from rebuilder 'https://mirror.notset.fr/qubes/rebuild/yum/r4.1/vm/'
Copy final product to verification directory
Load in-toto layout '/home/user/dnf-transport-in-toto/data/root.layout' (DNF global_info)
Load in-toto layout key(s) '['9fa64b92f95e706bf28e2ca6484010b5cdc576e2']' (DNF global_info)
Use gpg keyring '/home/user/dnf-transport-in-toto/data/gnupg' (DNF global_info)
Run in-toto verification
In-toto verification for 'qubes-u2f-1.2.8-1.fc33.noarch' passed! :)
Dependencies resolved.
=======================================================================================
Package Arch Version Repository Size
=======================================================================================
Installing:
qubes-u2f noarch 1.2.8-1.fc33 qubes-vm-r4.1-current-testing 264 k
Installing dependencies:
hidapi x86_64 0.9.0-4.fc33 fedora 45 k
python3-cryptography x86_64 3.2.1-2.fc33 updates 546 k
python3-hidapi x86_64 0.9.0.post2-2.fc33 fedora 50 k
python3-u2flib-host noarch 3.0.3-9.fc33 qubes-vm-r4.1-current-testing 49 k
Transaction Summary
=======================================================================================
Install 5 Packages
Total download size: 954 k
Installed size: 3.6 M
Next steps
As explained above, some parts still need finishing, as well as cleanups and proper documentation. But we are very close to the point where every Debian package we build can be independently verified. The very same tooling we’ve made can be used to verify native Debian packages, too, which should also be helpful for non-Qubes Debian users. Similar progress has already been made for Fedora, although some more work is needed on the Fedora side to allow reproducing native (not only Qubes-related) packages.
In the broader future, our ultimate goal is to make all parts of Qubes OS reproducible, including templates and the installation image. Reproducible packages are the first step toward this goal, which incidentally is also the most valuable step to our users and the broader community.
Acknowledgements
This work is possible thanks to generous support (https://www.qubes-os.org/news/2020/05/22/moss-mission-partners-award/) from Mozilla Open Source Support (MOSS) (https://www.mozilla.org/en-US/moss/).
I’d like also to thank GitLab for granting us a free GitLab Gold license, which enabled much higher service quotas.
Copy final product to verification directory
Load in-toto layout '/home/user/dnf-transport-in-toto/data/root.layout' (DNF global_info)
Load in-toto layout key(s) '['9fa64b92f95e706bf28e2ca6484010b5cdc576e2']' (DNF global_info)
Use gpg keyring '/home/user/dnf-transport-in-toto/data/gnupg' (DNF global_info)
Run in-toto verification
In-toto verification for 'qubes-u2f-1.2.8-1.fc33.noarch' passed! :)
Dependencies resolved.
=======================================================================================
Package Arch Version Repository Size
=======================================================================================
Installing:
qubes-u2f noarch 1.2.8-1.fc33 qubes-vm-r4.1-current-testing 264 k
Installing dependencies:
hidapi x86_64 0.9.0-4.fc33 fedora 45 k
python3-cryptography x86_64 3.2.1-2.fc33 updates 546 k
python3-hidapi x86_64 0.9.0.post2-2.fc33 fedora 50 k
python3-u2flib-host noarch 3.0.3-9.fc33 qubes-vm-r4.1-current-testing 49 k
Transaction Summary
=======================================================================================
Install 5 Packages
Total download size: 954 k
Installed size: 3.6 M
Next steps
As explained above, some parts still need finishing, as well as cleanups and proper documentation. But we are very close to the point where every Debian package we build can be independently verified. The very same tooling we’ve made can be used to verify native Debian packages, too, which should also be helpful for non-Qubes Debian users. Similar progress has already been made for Fedora, although some more work is needed on the Fedora side to allow reproducing native (not only Qubes-related) packages.
In the broader future, our ultimate goal is to make all parts of Qubes OS reproducible, including templates and the installation image. Reproducible packages are the first step toward this goal, which incidentally is also the most valuable step to our users and the broader community.
Acknowledgements
This work is possible thanks to generous support (https://www.qubes-os.org/news/2020/05/22/moss-mission-partners-award/) from Mozilla Open Source Support (MOSS) (https://www.mozilla.org/en-US/moss/).
I’d like also to thank GitLab for granting us a free GitLab Gold license, which enabled much higher service quotas.
Xen 4.15 RC1 – Please test
https://xenproject.org/2021/03/02/xen-4-15-rc1-please-test/
Xen 4.15 is in code freeze, and we cut RC1 yesterday. Please help us test it to make sure Xen 4.15 is a high quality release (and that it works...
https://xenproject.org/2021/03/02/xen-4-15-rc1-please-test/
Xen 4.15 is in code freeze, and we cut RC1 yesterday. Please help us test it to make sure Xen 4.15 is a high quality release (and that it works...