While doing the data crunching, I was a bit fascinated by three large groups of people: those from capital cities just putting down the name of the capital (omitting the country name), people in the US replying with just the name of their town (I’ve learned a lot about small American towns!) and people in the UK clarifying they are not English, thank you very much.
I had to smile at “United Kingdom of England and Some Actually Good Countries”.
We’re very interested in the hardware people are using and want to use with Qubes. Hardware is always a difficult subject for us, as there’s a lot of possible combinations and not nearly enough manpower to test and fix bugs for all of them, and we want to know where to focus our resources.
This intuition was well confirmed by the survey: hardware compatibility was something a lot of people mentioned in the “reasons for not using Qubes/reasons for stopping using Qubes” questions.
Following the common trend in modern hardware, most people use laptops or laptops and desktops equally (only 22% of our respondents use mostly a desktop computer), and most Qubes users tend to use it on a laptop (63% of them in the survey).
A lot of people use external monitors with their laptops (over 55% of laptop users), and we know an external monitor can be tricky to use with Qubes, leading to all sorts of annoying problems with layout or input detection. (If you haven’t yet tried it, take a look here: Qubes GUI Troubleshooting (https://www.qubes-os.org/doc/gui-troubleshooting/)).
A significant number of respondents also say they use cameras (36%) and microphones (60%). It makes me wonder what the responses to this question would be a year ago, before so many of us started working remotely.
As far as desired Qubes localization goes, there were few surprises, with the overwhelming majority preferring English (for a survey in English, it’s hard to be shocked by this result), and the next places being taken by German (over 200 votes), French (over 120 votes), Spanish (over 80) and Russian (over 70).
One impressive polyglot said they use a different language in each AppVM to easily distinguish their working environments, and I have to say, I wish I spoke enough languages to achieve that!
We asked about the OS our respondents find most comfortable, and, clearly, most prefer using Linux (48%), with Windows and Qubes (about 21% each) close seconds.
Finally, there’s little love for Mac OS, with less than 10% of respondents listing it as “most comfortable”.
Among Linux users, the range of distributions wasn’t very surprising, with Debian and Ubuntu as clear leaders, with over 50% selecting each as the distribution they use.
We also asked about the distribution you would prefer as the default template for AppVMs.
Debian got over twice as many votes (686) as the runner-up, Fedora (336).
Sounds like a good moment to mention that in Qubes 4.1 you can choose the default template at install (currently between Fedora and Debian).
Arch Linux (third place, with 74 people writing it in as ‘Other’) is also available as a community template and is well-maintained.
Interestingly enough, just using a distribution doesn’t mean someone wants to use it as the default template in Qubes, with some distributions having much more ardent supporters.
82% of people who use Debian want it as the default template, which is not that surprising, as Debian was one of the options explicitly offered in the “default distro” question.
But also almost 50% of NixOS users want it as the default template, which even from a purely methodological point of view is a lot, as they had to explicitly write this distribution down in the second question.
NixOS has some very devoted users!
On the other hand, although Ubuntu was one of the most popular distributions, only 4% of its users wrote it as their preferred default distribution…
Distribution
Users
Want it as default template
Debian
1103
82%
Ubuntu
I had to smile at “United Kingdom of England and Some Actually Good Countries”.
We’re very interested in the hardware people are using and want to use with Qubes. Hardware is always a difficult subject for us, as there’s a lot of possible combinations and not nearly enough manpower to test and fix bugs for all of them, and we want to know where to focus our resources.
This intuition was well confirmed by the survey: hardware compatibility was something a lot of people mentioned in the “reasons for not using Qubes/reasons for stopping using Qubes” questions.
Following the common trend in modern hardware, most people use laptops or laptops and desktops equally (only 22% of our respondents use mostly a desktop computer), and most Qubes users tend to use it on a laptop (63% of them in the survey).
A lot of people use external monitors with their laptops (over 55% of laptop users), and we know an external monitor can be tricky to use with Qubes, leading to all sorts of annoying problems with layout or input detection. (If you haven’t yet tried it, take a look here: Qubes GUI Troubleshooting (https://www.qubes-os.org/doc/gui-troubleshooting/)).
A significant number of respondents also say they use cameras (36%) and microphones (60%). It makes me wonder what the responses to this question would be a year ago, before so many of us started working remotely.
As far as desired Qubes localization goes, there were few surprises, with the overwhelming majority preferring English (for a survey in English, it’s hard to be shocked by this result), and the next places being taken by German (over 200 votes), French (over 120 votes), Spanish (over 80) and Russian (over 70).
One impressive polyglot said they use a different language in each AppVM to easily distinguish their working environments, and I have to say, I wish I spoke enough languages to achieve that!
We asked about the OS our respondents find most comfortable, and, clearly, most prefer using Linux (48%), with Windows and Qubes (about 21% each) close seconds.
Finally, there’s little love for Mac OS, with less than 10% of respondents listing it as “most comfortable”.
Among Linux users, the range of distributions wasn’t very surprising, with Debian and Ubuntu as clear leaders, with over 50% selecting each as the distribution they use.
We also asked about the distribution you would prefer as the default template for AppVMs.
Debian got over twice as many votes (686) as the runner-up, Fedora (336).
Sounds like a good moment to mention that in Qubes 4.1 you can choose the default template at install (currently between Fedora and Debian).
Arch Linux (third place, with 74 people writing it in as ‘Other’) is also available as a community template and is well-maintained.
Interestingly enough, just using a distribution doesn’t mean someone wants to use it as the default template in Qubes, with some distributions having much more ardent supporters.
82% of people who use Debian want it as the default template, which is not that surprising, as Debian was one of the options explicitly offered in the “default distro” question.
But also almost 50% of NixOS users want it as the default template, which even from a purely methodological point of view is a lot, as they had to explicitly write this distribution down in the second question.
NixOS has some very devoted users!
On the other hand, although Ubuntu was one of the most popular distributions, only 4% of its users wrote it as their preferred default distribution…
Distribution
Users
Want it as default template
Debian
1103
82%
Ubuntu
928
4%
Fedora
783
55%
Arch Linux
438
23%
CentOS
265
6%
Gentoo
86
34%
NixOS
46
46%
(This table contains only the most popular choices, not all answers.)
From a UX development point of view, a particularly important question for us was “How many qubes do you typically run at the same time?”
Turns out that about the same number of people run 3-5 qubes as 6-10 (about 38%).
This will definitely be a huge help in future development of the various Qubes tools and widgets. It’s also a bit more than we suspected before!
4%
Fedora
783
55%
Arch Linux
438
23%
CentOS
265
6%
Gentoo
86
34%
NixOS
46
46%
(This table contains only the most popular choices, not all answers.)
From a UX development point of view, a particularly important question for us was “How many qubes do you typically run at the same time?”
Turns out that about the same number of people run 3-5 qubes as 6-10 (about 38%).
This will definitely be a huge help in future development of the various Qubes tools and widgets. It’s also a bit more than we suspected before!
As far the “how did you learn about Qubes” questions go, I think the conclusion on my side (as one of the authors of the survey) is simple: I really should have just included a “from Edward Snowden” option there, which would have saved our respondents some typing!
The survey covered many more questions than are described above, and they are all important for us as a team to learn more about you, our users, to know what to focus on and what to work on, what will work best and what may not be a great idea.
Don’t worry. If a question isn’t discussed above, we have still read it!
And again, thank you everyone for participating (and the many, many kind words you shared in your surveys).
The survey covered many more questions than are described above, and they are all important for us as a team to learn more about you, our users, to know what to focus on and what to work on, what will work best and what may not be a great idea.
Don’t worry. If a question isn’t discussed above, we have still read it!
And again, thank you everyone for participating (and the many, many kind words you shared in your surveys).
Qubes Canary 025
https://www.qubes-os.org/news/2020/12/12/canary-025/
We have published Qubes Canary 025. The text of this canary is
reproduced below.
Note: We have decided to make some minor formatting changes to the way
Qubes Canary and Qubes Security Bulletin (QSB) numbers are printed,
such as dropping the ‘#’ symbol and using hyphens instead of spaces.
This canary and its accompanying signatures will always be available in
the Qubes Security Pack (qubes-secpack).
View Qubes Canary 025 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/canaries/canary-025-2020.txt
Learn about the qubes-secpack, including how to obtain, verify, and
read it:
https://www.qubes-os.org/security/pack/
View all past canaries:
https://www.qubes-os.org/security/canaries/
---===[ Qubes Canary 025 ]===---
Statements
-----------
The Qubes core developers who have digitally signed this file [1]
state the following:
1. The date of issue of this canary is December 8, 2020.
2. There have been 62 Qubes Security Bulletins published so far.
3. The Qubes Master Signing Key fingerprint is:
427F 11FD 0FAA 4B08 0123 F01C DDFA 1A3E 3687 9494
4. No warrants have ever been served to us with regard to the Qubes OS
Project (e.g. to hand out the private signing keys or to introduce
backdoors).
5. We plan to publish the next of these canary statements in the first
two weeks of March 2020. Special note should be taken if no new canary
is published by that time or if the list of statements changes without
plausible explanation.
Special announcements
----------------------
None.
Disclaimers and notes
----------------------
We would like to remind you that Qubes OS has been designed under the
assumption that all relevant infrastructure is permanently
compromised. This means that we assume NO trust in any of the servers
or services which host or provide any Qubes-related data, in
particular, software updates, source code repositories, and Qubes ISO
downloads.
This canary scheme is not infallible. Although signing the declaration
makes it very difficult for a third party to produce arbitrary
declarations, it does not prevent them from using force or other
means, like blackmail or compromising the signers' laptops, to coerce
us to produce false declarations.
The news feeds quoted below (Proof of freshness) serves to demonstrate
that this canary could not have been created prior to the date stated.
It shows that a series of canaries was not created in advance.
This declaration is merely a best effort and is provided without any
guarantee or warranty. It is not legally binding in any way to
anybody. None of the signers should be ever held legally responsible
for any of the statements made here.
Proof of freshness
-------------------
Tue, 08 Dec 2020 16:46:42 +0000
Source: DER SPIEGEL - International (https://www.spiegel.de/international/index.rss)
Dangerous Accusations: German Tennis Star Alexander Zverev Faces Career Turning Point
Skiing in the Pandemic: Alpine Rivalries Flare amid Resort Closures
Biden's Goal of Saving the Iran Deal Just Got Harder - A Murder and an Ultimatum
Heiko Maas: Germany's Foreign Minister on the Future of Trans-Atlantic Relations
Generation Corona: The Pandemic Is Changing Our Children's Lives for the Worse
Source: NYT > World News (https://rss.nytimes.com/services/xml/rss/nyt/World.xml)
Covid-19 Live Updates: Britain Begins Vaccinating Citizens
U.K. Covid Vaccine: Side Effects, Safety, and Who Gets It First
U.S. Leaves Behind Afghan Bases — and a Legacy of Land Disputes
Covid Infections, and Blame, Rise Along Southeast Asian Borders
U.S. Imposes Sanctions on Chinese Officials Over Hong Kong Crackdown
Source: BBC News - World (https://feeds.bbci.co.uk/news/world/rss.xml)
Safety data on Pfizer jab released by US
Lloyd Austin: Biden picks ex-general as defence secretary
The man saving monkeys in the Colombian Amazon
https://www.qubes-os.org/news/2020/12/12/canary-025/
We have published Qubes Canary 025. The text of this canary is
reproduced below.
Note: We have decided to make some minor formatting changes to the way
Qubes Canary and Qubes Security Bulletin (QSB) numbers are printed,
such as dropping the ‘#’ symbol and using hyphens instead of spaces.
This canary and its accompanying signatures will always be available in
the Qubes Security Pack (qubes-secpack).
View Qubes Canary 025 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/canaries/canary-025-2020.txt
Learn about the qubes-secpack, including how to obtain, verify, and
read it:
https://www.qubes-os.org/security/pack/
View all past canaries:
https://www.qubes-os.org/security/canaries/
---===[ Qubes Canary 025 ]===---
Statements
-----------
The Qubes core developers who have digitally signed this file [1]
state the following:
1. The date of issue of this canary is December 8, 2020.
2. There have been 62 Qubes Security Bulletins published so far.
3. The Qubes Master Signing Key fingerprint is:
427F 11FD 0FAA 4B08 0123 F01C DDFA 1A3E 3687 9494
4. No warrants have ever been served to us with regard to the Qubes OS
Project (e.g. to hand out the private signing keys or to introduce
backdoors).
5. We plan to publish the next of these canary statements in the first
two weeks of March 2020. Special note should be taken if no new canary
is published by that time or if the list of statements changes without
plausible explanation.
Special announcements
----------------------
None.
Disclaimers and notes
----------------------
We would like to remind you that Qubes OS has been designed under the
assumption that all relevant infrastructure is permanently
compromised. This means that we assume NO trust in any of the servers
or services which host or provide any Qubes-related data, in
particular, software updates, source code repositories, and Qubes ISO
downloads.
This canary scheme is not infallible. Although signing the declaration
makes it very difficult for a third party to produce arbitrary
declarations, it does not prevent them from using force or other
means, like blackmail or compromising the signers' laptops, to coerce
us to produce false declarations.
The news feeds quoted below (Proof of freshness) serves to demonstrate
that this canary could not have been created prior to the date stated.
It shows that a series of canaries was not created in advance.
This declaration is merely a best effort and is provided without any
guarantee or warranty. It is not legally binding in any way to
anybody. None of the signers should be ever held legally responsible
for any of the statements made here.
Proof of freshness
-------------------
Tue, 08 Dec 2020 16:46:42 +0000
Source: DER SPIEGEL - International (https://www.spiegel.de/international/index.rss)
Dangerous Accusations: German Tennis Star Alexander Zverev Faces Career Turning Point
Skiing in the Pandemic: Alpine Rivalries Flare amid Resort Closures
Biden's Goal of Saving the Iran Deal Just Got Harder - A Murder and an Ultimatum
Heiko Maas: Germany's Foreign Minister on the Future of Trans-Atlantic Relations
Generation Corona: The Pandemic Is Changing Our Children's Lives for the Worse
Source: NYT > World News (https://rss.nytimes.com/services/xml/rss/nyt/World.xml)
Covid-19 Live Updates: Britain Begins Vaccinating Citizens
U.K. Covid Vaccine: Side Effects, Safety, and Who Gets It First
U.S. Leaves Behind Afghan Bases — and a Legacy of Land Disputes
Covid Infections, and Blame, Rise Along Southeast Asian Borders
U.S. Imposes Sanctions on Chinese Officials Over Hong Kong Crackdown
Source: BBC News - World (https://feeds.bbci.co.uk/news/world/rss.xml)
Safety data on Pfizer jab released by US
Lloyd Austin: Biden picks ex-general as defence secretary
The man saving monkeys in the Colombian Amazon
Charlie Hebdo attack: France seeks long jail terms in Paris trial
Christchurch massacre: Inquiry finds failures ahead of attack
Source: Blockchain.info
0000000000000000000c6550025327ca735099e0c621a9ad4599a49dab41f573
Footnotes
----------
[1] This file should be signed in two ways: (1) via detached PGP
signatures by each of the signers, distributed together with this
canary in the qubes-secpack.git repo, and (2) via digital signatures
on the corresponding qubes-secpack.git repo tags. [2]
[2] Don't just trust the contents of this file blindly! Verify the
digital signatures!
Christchurch massacre: Inquiry finds failures ahead of attack
Source: Blockchain.info
0000000000000000000c6550025327ca735099e0c621a9ad4599a49dab41f573
Footnotes
----------
[1] This file should be signed in two ways: (1) via detached PGP
signatures by each of the signers, distributed together with this
canary in the qubes-secpack.git repo, and (2) via digital signatures
on the corresponding qubes-secpack.git repo tags. [2]
[2] Don't just trust the contents of this file blindly! Verify the
digital signatures!
HVMI: Security Solutions Thrive on Friendly Hypervisors
https://xenproject.org/2020/12/14/hvmi-security-solutions-thrive-on-friendly-hypervisors/
This talk was given by Raul Tosa & Daniel Ticle, Bitdefender at the Xen Developer and Design Summit in July 2020. In July, Bitdefender open sourced Hypervisor Memory Introspection (HVMI)....
https://xenproject.org/2020/12/14/hvmi-security-solutions-thrive-on-friendly-hypervisors/
This talk was given by Raul Tosa & Daniel Ticle, Bitdefender at the Xen Developer and Design Summit in July 2020. In July, Bitdefender open sourced Hypervisor Memory Introspection (HVMI)....
QSB-063: Multiple Xen issues (XSA-115, XSA-325, XSA-350)
https://www.qubes-os.org/news/2020/12/16/qsb-063/
We have just published Qubes Security Bulletin (QSB) 063:
Stack corruption from XSA-346 change (XSA-355).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-063 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-063-2020.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View all XSAs mentioned in this QSB in the XSA Tracker:
https://www.qubes-os.org/security/xsa/
---===[ Qubes Security Bulletin 063 ]===---
2020-12-15
Multiple Xen issues (XSA-115, XSA-325, XSA-350)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Xen packages, version 4.8.5-28
- Linux kernel packages, versions 5.9.14-1, 5.4.83-1, 4.19.163-1
For Qubes 4.1:
- Xen packages, version 4.14.0-9
- Linux kernel packages, versions 5.9.14-1, 5.4.83-1, 4.19.163-1
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Xen binaries.
Summary
========
On 2020-12-15, the Xen Security Team published the following Xen
Security Advisories (XSAs):
XSA-115 [1] "xenstore watch notifications lacking permission checks"
| Neither xenstore implementation does any permissions checks when
| reporting a xenstore watch event.
|
| A guest administrator can watch the root xenstored node, which will
| cause notifications for every created, modified and deleted key.
|
| A guest administrator can also use the special watches, which will
| cause a notification every time a domain is created and destroyed.
|
| Data may include:
| - number, type and domids of other VMs
| - existence and domids of driver domains
| - numbers of virtual interfaces, block devices, vcpus
| - existence of virtual framebuffers and their backend style (eg,
| existence of VNC service)
| - Xen VM UUIDs for other domains
| - timing information about domain creation and device setup
| - some hints at the backend provisioning of VMs and their devices
|
| The watch events do not contain values stored in xenstore, only key
| names.
XSA-325 [2] "Xenstore: guests can disturb domain cleanup"
| Xenstored and guests communicate via a shared memory page using a
| specific protocol. When a guest violates this protocol, xenstored will
| drop the connection to that guest.
|
| Unfortunately this is done by just removing the guest from xenstored's
| internal management, resulting in the same actions as if the guest had
| been destroyed, including sending an @releaseDomain event.
|
| @releaseDomain events do not say guest has been removed. All watchers
| of this event must look at the states of all guests to find the guest
| which has been removed. When an @releaseDomain is generated due to
| domain xenstored protocol violation, As the guest is still running, so
| the watchers will not react.
|
| Later, when the guest is actually destroyed, xenstored will no longer
https://www.qubes-os.org/news/2020/12/16/qsb-063/
We have just published Qubes Security Bulletin (QSB) 063:
Stack corruption from XSA-346 change (XSA-355).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-063 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-063-2020.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View all XSAs mentioned in this QSB in the XSA Tracker:
https://www.qubes-os.org/security/xsa/
---===[ Qubes Security Bulletin 063 ]===---
2020-12-15
Multiple Xen issues (XSA-115, XSA-325, XSA-350)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Xen packages, version 4.8.5-28
- Linux kernel packages, versions 5.9.14-1, 5.4.83-1, 4.19.163-1
For Qubes 4.1:
- Xen packages, version 4.14.0-9
- Linux kernel packages, versions 5.9.14-1, 5.4.83-1, 4.19.163-1
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Xen binaries.
Summary
========
On 2020-12-15, the Xen Security Team published the following Xen
Security Advisories (XSAs):
XSA-115 [1] "xenstore watch notifications lacking permission checks"
| Neither xenstore implementation does any permissions checks when
| reporting a xenstore watch event.
|
| A guest administrator can watch the root xenstored node, which will
| cause notifications for every created, modified and deleted key.
|
| A guest administrator can also use the special watches, which will
| cause a notification every time a domain is created and destroyed.
|
| Data may include:
| - number, type and domids of other VMs
| - existence and domids of driver domains
| - numbers of virtual interfaces, block devices, vcpus
| - existence of virtual framebuffers and their backend style (eg,
| existence of VNC service)
| - Xen VM UUIDs for other domains
| - timing information about domain creation and device setup
| - some hints at the backend provisioning of VMs and their devices
|
| The watch events do not contain values stored in xenstore, only key
| names.
XSA-325 [2] "Xenstore: guests can disturb domain cleanup"
| Xenstored and guests communicate via a shared memory page using a
| specific protocol. When a guest violates this protocol, xenstored will
| drop the connection to that guest.
|
| Unfortunately this is done by just removing the guest from xenstored's
| internal management, resulting in the same actions as if the guest had
| been destroyed, including sending an @releaseDomain event.
|
| @releaseDomain events do not say guest has been removed. All watchers
| of this event must look at the states of all guests to find the guest
| which has been removed. When an @releaseDomain is generated due to
| domain xenstored protocol violation, As the guest is still running, so
| the watchers will not react.
|
| Later, when the guest is actually destroyed, xenstored will no longer
| have it stored in its internal data base, so no further @releaseDomain
| event will be sent. This can lead to a zombie domain; memory mappings
| of that guest's memory will not be removed, due to the missing
| event. This zombie domain will be cleaned up only after another domain
| is destroyed, as that will trigger another @releaseDomain event.
|
| If the device model of the guest which violated the Xenstore protocol
| is running in a stub-domain, a use-after-free case could happen in
| xenstored, after having removed the guest from its internal data base,
| possibly resulting in a crash of xenstored.
XSA-350 [3] "Use after free triggered by block frontend in Linux blkback"
| The Linux kernel PV block backend expects the kernel thread handler
| to reset ring->xenblkd to NULL when stopped. However, the handler may
| not have time to run if the frontend quickly toggle between the states
| connect and disconnect.
|
| As a consequence, the block backend may re-use a pointer after it was
| freed.
Impact
=======
XSA-115, as described by Xen Security Team:
| A guest administrator can observe non-sensitive domain and device
| lifecycle events relating to other guests. This information allows
| some insight into overall system configuration (including number of
| general nature of other guests), and configuration of other guests
| (including number and general nature of other guests' devices). This
| information might be commercially interesting or might make other
| attacks easier.
|
| There is not believed to be exposure of sensitive data. Specifically,
| there is no exposure of: VNC passwords; port numbers; pathnames in host
| and guest filesystems; cryptopgraphic keys; or within-guest data.
XSA-325:
This issue allows a compromised domain to delay resource cleanup,
including disposing DisposableVMs. The delay will be at most until
another domain shutdown. If an HVM domain (like sys-net or sys-usb)
triggers it, it may cause a use-after-free issue in the xenstored
daemon. It may cause the daemon to crash (preventing most further
operations in the system). Beyond DoS, it is unlikely that this
vulnerability could be exploited to compromise the system, but we
cannot completely rule out the possibility.
XSA-350, as described by Xen Security Team:
| A misbehaving guest can trigger a dom0 crash by continuously
| connecting / disconnecting a block frontend. Privileged escalation and
| information leak cannot be ruled out.
Credits
========
See the original Xen Security Advisories.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-115.html
[2] https://xenbits.xen.org/xsa/advisory-325.html
[3] https://xenbits.xen.org/xsa/advisory-350.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
| event will be sent. This can lead to a zombie domain; memory mappings
| of that guest's memory will not be removed, due to the missing
| event. This zombie domain will be cleaned up only after another domain
| is destroyed, as that will trigger another @releaseDomain event.
|
| If the device model of the guest which violated the Xenstore protocol
| is running in a stub-domain, a use-after-free case could happen in
| xenstored, after having removed the guest from its internal data base,
| possibly resulting in a crash of xenstored.
XSA-350 [3] "Use after free triggered by block frontend in Linux blkback"
| The Linux kernel PV block backend expects the kernel thread handler
| to reset ring->xenblkd to NULL when stopped. However, the handler may
| not have time to run if the frontend quickly toggle between the states
| connect and disconnect.
|
| As a consequence, the block backend may re-use a pointer after it was
| freed.
Impact
=======
XSA-115, as described by Xen Security Team:
| A guest administrator can observe non-sensitive domain and device
| lifecycle events relating to other guests. This information allows
| some insight into overall system configuration (including number of
| general nature of other guests), and configuration of other guests
| (including number and general nature of other guests' devices). This
| information might be commercially interesting or might make other
| attacks easier.
|
| There is not believed to be exposure of sensitive data. Specifically,
| there is no exposure of: VNC passwords; port numbers; pathnames in host
| and guest filesystems; cryptopgraphic keys; or within-guest data.
XSA-325:
This issue allows a compromised domain to delay resource cleanup,
including disposing DisposableVMs. The delay will be at most until
another domain shutdown. If an HVM domain (like sys-net or sys-usb)
triggers it, it may cause a use-after-free issue in the xenstored
daemon. It may cause the daemon to crash (preventing most further
operations in the system). Beyond DoS, it is unlikely that this
vulnerability could be exploited to compromise the system, but we
cannot completely rule out the possibility.
XSA-350, as described by Xen Security Team:
| A misbehaving guest can trigger a dom0 crash by continuously
| connecting / disconnecting a block frontend. Privileged escalation and
| information leak cannot be ruled out.
Credits
========
See the original Xen Security Advisories.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-115.html
[2] https://xenbits.xen.org/xsa/advisory-325.html
[3] https://xenbits.xen.org/xsa/advisory-350.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
Cut your Cloud Computing Costs by Half with Unikraft
https://xenproject.org/2020/12/18/cut-your-cloud-computing-costs-by-half-with-unikraft/
This post originally appeared on Linux.com A novel modular unikernel allows for extreme tailoring of your operating system to your application’s needs. A proof of concept, built on Unikraft, a...
https://xenproject.org/2020/12/18/cut-your-cloud-computing-costs-by-half-with-unikraft/
This post originally appeared on Linux.com A novel modular unikernel allows for extreme tailoring of your operating system to your application’s needs. A proof of concept, built on Unikraft, a...
XSAs released on 2021-01-21
https://www.qubes-os.org/news/2021/01/22/xsas-released-on-2021-01-21/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-01-21.
The security of Qubes OS is not affected by these XSAs.
Therefore, user action is not required.
XSAs that affect the security of Qubes OS (user action required)
None.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
XSA-360 (DoS only)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
https://www.qubes-os.org/news/2021/01/22/xsas-released-on-2021-01-21/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-01-21.
The security of Qubes OS is not affected by these XSAs.
Therefore, user action is not required.
XSAs that affect the security of Qubes OS (user action required)
None.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
XSA-360 (DoS only)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
The Xen Project at FOSDEM: Unikraft and XCP-ng
https://xenproject.org/2021/02/04/the-xen-project-at-fosdem-unikraft-and-xcp-ng/
It’s that time of year when open source community gathers for FOSDEM! FOSDEM is a free event for software developers to meet, share ideas, and collaborate. This year’s event will...
https://xenproject.org/2021/02/04/the-xen-project-at-fosdem-unikraft-and-xcp-ng/
It’s that time of year when open source community gathers for FOSDEM! FOSDEM is a free event for software developers to meet, share ideas, and collaborate. This year’s event will...
2021: Xen Project, virtualization and beyond
https://xenproject.org/2021/02/10/2021-xen-project-virtualization-and-beyond/
This post originally appeared on VM Blog. By George Dunlap, Advisory Board Chair for the Xen Project The Xen Project has been around for the better part of two decades. As...
https://xenproject.org/2021/02/10/2021-xen-project-virtualization-and-beyond/
This post originally appeared on VM Blog. By George Dunlap, Advisory Board Chair for the Xen Project The Xen Project has been around for the better part of two decades. As...
QSB-064: Linux: error handling issues in blkback's grant mapping (XSA-365)
https://www.qubes-os.org/news/2021/02/17/qsb-064/
We have just published Qubes Security Bulletin (QSB) 064:
Linux: error handling issues in blkback’s grant mapping (XSA-365).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-064 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-064-2021.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View XSA-365 in the XSA Tracker:
https://www.qubes-os.org/security/xsa/#365
---===[ Qubes Security Bulletin 064 ]===---
2021-02-16
Linux: error handling issues in blkback's grant mapping (XSA-365)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Linux kernel packages, versions 5.10.16-1, 5.4.98-1, 4.19.176-1
For Qubes 4.1:
- Linux kernel packages, versions 5.10.16-1, 5.4.98-1
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Linux kernel binaries.
Summary
========
On 2021-02-16, the Xen Security Team published the following Xen
Security Advisory (XSA):
XSA-365 [1] "Linux: error handling issues in blkback's grant mapping"
| To service requests, the driver maps grant references provided by the
| frontend. In this process, errors may be encountered. In one case an
| error encountered earlier might be discarded by later processing,
| resulting in the caller assuming successful mapping, and hence
| subsequent operations trying to access space that wasn't mapped. In
| another case internal state would be insufficiently updated, preventing
| safe recovery from the error.
Impact
=======
XSA-365, as described by Xen Security Team:
| A malicious or buggy frontend driver may be able to crash the
| corresponding backend driver, potentially affecting the entire domain
| running the backend driver. In configurations without driver domains
| or similar disaggregation, that is a host-wide denial of sevice.
|
| Privilege escalation and information leaks cannot be ruled out.
Credits
========
See the original Xen Security Advisories.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-365.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
https://www.qubes-os.org/news/2021/02/17/qsb-064/
We have just published Qubes Security Bulletin (QSB) 064:
Linux: error handling issues in blkback’s grant mapping (XSA-365).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-064 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-064-2021.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View XSA-365 in the XSA Tracker:
https://www.qubes-os.org/security/xsa/#365
---===[ Qubes Security Bulletin 064 ]===---
2021-02-16
Linux: error handling issues in blkback's grant mapping (XSA-365)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Linux kernel packages, versions 5.10.16-1, 5.4.98-1, 4.19.176-1
For Qubes 4.1:
- Linux kernel packages, versions 5.10.16-1, 5.4.98-1
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Linux kernel binaries.
Summary
========
On 2021-02-16, the Xen Security Team published the following Xen
Security Advisory (XSA):
XSA-365 [1] "Linux: error handling issues in blkback's grant mapping"
| To service requests, the driver maps grant references provided by the
| frontend. In this process, errors may be encountered. In one case an
| error encountered earlier might be discarded by later processing,
| resulting in the caller assuming successful mapping, and hence
| subsequent operations trying to access space that wasn't mapped. In
| another case internal state would be insufficiently updated, preventing
| safe recovery from the error.
Impact
=======
XSA-365, as described by Xen Security Team:
| A malicious or buggy frontend driver may be able to crash the
| corresponding backend driver, potentially affecting the entire domain
| running the backend driver. In configurations without driver domains
| or similar disaggregation, that is a host-wide denial of sevice.
|
| Privilege escalation and information leaks cannot be ruled out.
Credits
========
See the original Xen Security Advisories.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-365.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
XSAs released on 2021-02-16
https://www.qubes-os.org/news/2021/02/17/xsas-released-on-2021-02-16/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-02-16.
The security of Qubes OS is affected by one or more of these XSAs.
Therefore, user action is required.
XSAs that affect the security of Qubes OS (user action required)
The following XSAs do affect the security of Qubes OS:
XSA-365
Please see QSB-064 (https://www.qubes-os.org/news/2021/02/17/qsb-064/) for the actions users must take in order to protect themselves, as well as further details about these XSAs.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
XSA-361 (DoS-only)
XSA-362 (DoS-only)
XSA-363 (DoS-only)
XSA-364 (ARM-only)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
https://www.qubes-os.org/news/2021/02/17/xsas-released-on-2021-02-16/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-02-16.
The security of Qubes OS is affected by one or more of these XSAs.
Therefore, user action is required.
XSAs that affect the security of Qubes OS (user action required)
The following XSAs do affect the security of Qubes OS:
XSA-365
Please see QSB-064 (https://www.qubes-os.org/news/2021/02/17/qsb-064/) for the actions users must take in order to protect themselves, as well as further details about these XSAs.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
XSA-361 (DoS-only)
XSA-362 (DoS-only)
XSA-363 (DoS-only)
XSA-364 (ARM-only)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
QSB-065: Missed flush in XSA-321 backport (XSA-366)
https://www.qubes-os.org/news/2021/02/19/qsb-065/
We have just published Qubes Security Bulletin (QSB) 065:
Missed flush in XSA-321 backport (XSA-366).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-065 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-065-2021.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View XSA-366 in the XSA Tracker:
https://www.qubes-os.org/security/xsa/#366
---===[ Qubes Security Bulletin 065 ]===---
2021-02-18
Missed flush in XSA-321 backport (XSA-366)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Xen packages, versions 4.8.5-30
For Qubes 4.1: not affected
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Xen binaries.
Summary
========
On 2021-02-18, the Xen Security Team published the following Xen
Security Advisory (XSA):
XSA-366 [1] "missed flush in XSA-321 backport"
| An oversight was made when backporting XSA-320, leading entries in the
| IOMMU not being properly updated under certain circumstances.
Impact
=======
XSA-366, as described by the Xen Security Team:
| A malicious guest may be able to retain read/write DMA access to
| frames returned to Xen's free pool, and later reused for another
| purpose. Host crashes (leading to a Denial of Service) and privilege
| escalation cannot be ruled out.
Credits
========
See the original Xen Security Advisory.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-366.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
https://www.qubes-os.org/news/2021/02/19/qsb-065/
We have just published Qubes Security Bulletin (QSB) 065:
Missed flush in XSA-321 backport (XSA-366).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-065 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-065-2021.txt
Learn about the qubes-secpack, including how to obtain, verify, and read it:
https://www.qubes-os.org/security/pack/
View all past QSBs:
https://www.qubes-os.org/security/bulletins/
View XSA-366 in the XSA Tracker:
https://www.qubes-os.org/security/xsa/#366
---===[ Qubes Security Bulletin 065 ]===---
2021-02-18
Missed flush in XSA-321 backport (XSA-366)
User action required
=====================
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0:
- Xen packages, versions 4.8.5-30
For Qubes 4.1: not affected
The packages are to be installed in dom0 via the Qube Manager or via
the qubes-dom0-update command as follows:
For updates from the stable repository (not immediately available):
$ sudo qubes-dom0-update
For updates from the security-testing repository:
$ sudo qubes-dom0-update --enablerepo=qubes-dom0-security-testing
A system restart will be required afterwards.
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Xen binaries.
Summary
========
On 2021-02-18, the Xen Security Team published the following Xen
Security Advisory (XSA):
XSA-366 [1] "missed flush in XSA-321 backport"
| An oversight was made when backporting XSA-320, leading entries in the
| IOMMU not being properly updated under certain circumstances.
Impact
=======
XSA-366, as described by the Xen Security Team:
| A malicious guest may be able to retain read/write DMA access to
| frames returned to Xen's free pool, and later reused for another
| purpose. Host crashes (leading to a Denial of Service) and privilege
| escalation cannot be ruled out.
Credits
========
See the original Xen Security Advisory.
References
===========
[1] https://xenbits.xen.org/xsa/advisory-366.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
XSAs released on 2021-02-18
https://www.qubes-os.org/news/2021/02/19/xsas-released-on-2021-02-18/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-02-18.
The security of Qubes OS is affected by one or more of these XSAs.
Therefore, user action is required.
XSAs that affect the security of Qubes OS (user action required)
The following XSAs do affect the security of Qubes OS:
XSA-366
Please see QSB-065 (https://www.qubes-os.org/news/2021/02/19/qsb-065/) for the actions users must take in order to protect themselves, as well as further details about these XSAs.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
(None)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
https://www.qubes-os.org/news/2021/02/19/xsas-released-on-2021-02-18/
The Xen Project released one or more new Xen Security Advisories (XSAs) on 2021-02-18.
The security of Qubes OS is affected by one or more of these XSAs.
Therefore, user action is required.
XSAs that affect the security of Qubes OS (user action required)
The following XSAs do affect the security of Qubes OS:
XSA-366
Please see QSB-065 (https://www.qubes-os.org/news/2021/02/19/qsb-065/) for the actions users must take in order to protect themselves, as well as further details about these XSAs.
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
(None)
Related links
Qubes Security Pack (qubes-secpack) (https://www.qubes-os.org/security/pack/)
Qubes Security Bulletins (QSBs) (https://www.qubes-os.org/security/bulletins/)
XSA Tracker (https://www.qubes-os.org/security/xsa/)
Fedora 33 TemplateVMs available
https://www.qubes-os.org/news/2021/02/25/fedora-33-templates-available/
New Fedora 33 TemplateVMs are now available for both Qubes 4.0 and 4.1.
Important: If you wish to use the Qubes Update widget to update a
Fedora 33 template, you must first switch (https://www.qubes-os.org/doc/templates/#switching) the
default-mgmt-dvm qube to a Fedora 33 template. (Alternatively, you
can create a separate management DisposableVM Template based on a
Fedora 33 template for the purpose of updating Fedora 33 templates.)
This does not affect updating internally using dnf.
Instructions are available for upgrading Fedora TemplateVMs (https://www.qubes-os.org/doc/template/fedora/upgrade/). We also
provide fresh Fedora 33 TemplateVM packages through the official Qubes
repositories, which you can get with the following commands (in dom0).
Standard (https://www.qubes-os.org/doc/templates/fedora/) Fedora 33 TemplateVM:
$ sudo qubes-dom0-update qubes-template-fedora-33
Minimal (https://www.qubes-os.org/doc/templates/minimal/) Fedora 33 TemplateVM:
$ sudo qubes-dom0-update qubes-template-fedora-33-minimal
After installing or upgrading a TemplateVM, please remember to update (https://www.qubes-os.org/doc/software-update-domu/)
(see important note above) and switch all qubes that were using the
old template to use the new one (https://www.qubes-os.org/doc/templates/#switching).
https://www.qubes-os.org/news/2021/02/25/fedora-33-templates-available/
New Fedora 33 TemplateVMs are now available for both Qubes 4.0 and 4.1.
Important: If you wish to use the Qubes Update widget to update a
Fedora 33 template, you must first switch (https://www.qubes-os.org/doc/templates/#switching) the
default-mgmt-dvm qube to a Fedora 33 template. (Alternatively, you
can create a separate management DisposableVM Template based on a
Fedora 33 template for the purpose of updating Fedora 33 templates.)
This does not affect updating internally using dnf.
Instructions are available for upgrading Fedora TemplateVMs (https://www.qubes-os.org/doc/template/fedora/upgrade/). We also
provide fresh Fedora 33 TemplateVM packages through the official Qubes
repositories, which you can get with the following commands (in dom0).
Standard (https://www.qubes-os.org/doc/templates/fedora/) Fedora 33 TemplateVM:
$ sudo qubes-dom0-update qubes-template-fedora-33
Minimal (https://www.qubes-os.org/doc/templates/minimal/) Fedora 33 TemplateVM:
$ sudo qubes-dom0-update qubes-template-fedora-33-minimal
After installing or upgrading a TemplateVM, please remember to update (https://www.qubes-os.org/doc/software-update-domu/)
(see important note above) and switch all qubes that were using the
old template to use the new one (https://www.qubes-os.org/doc/templates/#switching).
Improvements in testing and building: GitLab CI and reproducible builds
https://www.qubes-os.org/news/2021/02/28/improvements-in-testing-and-building/
Over the last couple of months, we have made some significant changes to two important parts of the Qubes development process: testing and building.
What are continuous integration (CI) and reproducible builds?
Automated testing is a major part of the software development process. It spares developers many, many hours of manual testing that would still miss some bugs and other problems. In Qubes development, we’re using an approach called “continuous integration” (CI), in which local changes made by the developers are frequently merged and tested remotely, using dedicated automated testing solutions. This is very important both for maintaining consistent code quality (all changes are tested) and for making development easier for the developers. Testing Qubes is not easy. Since Qubes is an entire operating system, doing the testing on the same system in which you’re developing is a bit like building a rocket landing system en route to Mars — not impossible, but very stressful.
The second area of improvement is the build process. The term “reproducible builds (https://reproducible-builds.org/)” refers to a process in which the same source code always compiles into exactly the same binary (for example, a package used to install a program via a package manager like dnf or apt). Why is this difficult to achieve? After all, computers are not random. Shouldn’t builds be reproducible by default, without requiring special effort to make them deterministic? Unfortunately, it’s not that simple. There are thousands of variables influencing the way binaries are built, ranging from the time of day to the availability of remote servers and locale settings.
Ensuring that binaries are built the same way every time is surprisingly difficult. However, the effort is worth the security benefits. To understand these benefits, imagine that an attacker wishes to feed unsuspecting users a compromised package. The attacker knows that the source code is public, so any malicious code he inserts into it would be highly exposed and at risk of detection. On the other hand, he reasons, compromising the build infrastructure would allow him to surreptitiously insert malicious changes that would make it into the resultant package. Since the source code remains untouched, his malicious changes are less likely to be detected. This is where the value of reproducible builds comes in. If the build process is reproducible, then we will immediately notice that building a package from the untouched source code results in a package that is different from the compromised one. This would be a major red flag that would prompt an immediate security investigation.
GitLab-CI migration
As many of you are aware, we migrated from Travis-CI to GitLab-CI late last year. While the direct reason (https://www.mail-archive.com/qubes-devel@googlegroups.com/msg04698.html) was a change in the Travis-CI terms of service, GitLab-CI gives us many additional benefits. Just to name a few:
A modern execution environment with native Docker support: We can use whatever base environment we like. We are no longer constrained to specific (not so fresh) Ubuntu versions.
Much more flexible job definitions, including dependencies among them: We use this to split jobs into smaller pieces that can run in parallel and reduce duplication among them.
Out-of-the-box support for caching and artifacts: Another feature allowing for a great speed-up of our tests. A specific build environment can be stored with a pre-populated cache, for example avoiding the need to create a chroot environment each time.
Higher time limits and the ability to connect our own workers: This allows us to automatically test bigger components like the Linux kernel (which previously didn’t fit into Travis-CI’s hard time limit).
https://www.qubes-os.org/news/2021/02/28/improvements-in-testing-and-building/
Over the last couple of months, we have made some significant changes to two important parts of the Qubes development process: testing and building.
What are continuous integration (CI) and reproducible builds?
Automated testing is a major part of the software development process. It spares developers many, many hours of manual testing that would still miss some bugs and other problems. In Qubes development, we’re using an approach called “continuous integration” (CI), in which local changes made by the developers are frequently merged and tested remotely, using dedicated automated testing solutions. This is very important both for maintaining consistent code quality (all changes are tested) and for making development easier for the developers. Testing Qubes is not easy. Since Qubes is an entire operating system, doing the testing on the same system in which you’re developing is a bit like building a rocket landing system en route to Mars — not impossible, but very stressful.
The second area of improvement is the build process. The term “reproducible builds (https://reproducible-builds.org/)” refers to a process in which the same source code always compiles into exactly the same binary (for example, a package used to install a program via a package manager like dnf or apt). Why is this difficult to achieve? After all, computers are not random. Shouldn’t builds be reproducible by default, without requiring special effort to make them deterministic? Unfortunately, it’s not that simple. There are thousands of variables influencing the way binaries are built, ranging from the time of day to the availability of remote servers and locale settings.
Ensuring that binaries are built the same way every time is surprisingly difficult. However, the effort is worth the security benefits. To understand these benefits, imagine that an attacker wishes to feed unsuspecting users a compromised package. The attacker knows that the source code is public, so any malicious code he inserts into it would be highly exposed and at risk of detection. On the other hand, he reasons, compromising the build infrastructure would allow him to surreptitiously insert malicious changes that would make it into the resultant package. Since the source code remains untouched, his malicious changes are less likely to be detected. This is where the value of reproducible builds comes in. If the build process is reproducible, then we will immediately notice that building a package from the untouched source code results in a package that is different from the compromised one. This would be a major red flag that would prompt an immediate security investigation.
GitLab-CI migration
As many of you are aware, we migrated from Travis-CI to GitLab-CI late last year. While the direct reason (https://www.mail-archive.com/qubes-devel@googlegroups.com/msg04698.html) was a change in the Travis-CI terms of service, GitLab-CI gives us many additional benefits. Just to name a few:
A modern execution environment with native Docker support: We can use whatever base environment we like. We are no longer constrained to specific (not so fresh) Ubuntu versions.
Much more flexible job definitions, including dependencies among them: We use this to split jobs into smaller pieces that can run in parallel and reduce duplication among them.
Out-of-the-box support for caching and artifacts: Another feature allowing for a great speed-up of our tests. A specific build environment can be stored with a pre-populated cache, for example avoiding the need to create a chroot environment each time.
Higher time limits and the ability to connect our own workers: This allows us to automatically test bigger components like the Linux kernel (which previously didn’t fit into Travis-CI’s hard time limit).
We still host the actual code on GitHub. We use GitLab only for CI. This mode of operation is supported natively by GitLab, but this support is quite limited. Most importantly, it does not support (https://gitlab.com/gitlab-org/gitlab/-/issues/5667) testing pull requests made from repository forks, which is the vast majority of our pull requests (if not all of them). For this reason, Frédéric ended up creating our own integration (https://github.com/fepitre/qubes-g2g-continuous-integration), which takes care of uploading the code to GitLab, scheduling CI jobs, and reporting the results back to GitHub. In fact, we are not the only project that has taken this route. Several other similar projects, such as check_pr, coq bot, and labhub, have also done so.
In the process, our CI has gained some nice extra features:
Control via special comments in pull requests:
PipelineRetry — restarts a CI job
PipelineRefresh — refreshes job status in GitHub (should normally not be needed)
testdeploy — in a website-related pull request, takes the website built in GitLab-CI and publishes it under a temporary address (works only after the GitLab-CI build completes and only for selected users)
More tests, specifically for reproducible builds of packages
We also have a nice summary page (https://qubesos.gitlab.io/qubes-g2g-report/) with all the recent jobs states, which is updated daily.
Reproducible builds
Our current short-term goal for reproducible builds in Qubes OS is to integrate what is realistically possible in Debian. Specifically, we aim to get our Debian templates to the state in which packages can be installed only when configured rebuilders confirm they really came from the source code we publish. (A “rebuilder” is a program that takes a binary and its purported source code as inputs, along with any applicable metadata, and attempts to build an identical binary from the source code. The goal is to check whether the binary was really compiled from its claimed source code.) For this task, we focus specifically on Debian Bullseye (testing version), which includes most of the reproducibility work done by the Debian community.
To start, we had to bring our packages to the state in which they could be built reproducibly. GitLab-CI massively helped here with identifying issues. One of the CI jobs we added is running the reprotest tool on our packages. This tool builds the package twice in two different build environments and compares the results. If there are any differences, the job automatically runs the diffoscope tool, which provides a detailed, in-depth report.
Now, all our packages for Debian Bullseye are reproducible, and we are actively monitoring it with our CI. This is designated by the “reproducible” badge on the CI summary page (https://qubesos.gitlab.io/qubes-g2g-report/).
The next step on the journey is to enable an arbitrary user or organization to challenge the official packages — to take the sources, together with the buildinfo file describing how it was built and to verify that they result in the same binary package. Theoretically, there is a debrebuild.pl noscript in Debian’s devnoscripts package that does exactly this. We started by adding missing parts to it, but it turned out there were several limitations making it unsuitable for our use case. The primary limitation is support for only one source repository. (We need two: original Debian and Qubes.) Since debrebuild.pl, as the name suggests, is written in Perl, it isn’t very friendly for contributing to. For these reasons, we decided to write our own similar tool in Python. While this was a bit of work, the end result (https://github.com/fepitre/debrebuild) is very nice: about 900 lines of code, with proper structure using classes. We are in the process of upstreaming this code back to Debian, potentially to replace the original debrebuild.pl tool.
In the process, our CI has gained some nice extra features:
Control via special comments in pull requests:
PipelineRetry — restarts a CI job
PipelineRefresh — refreshes job status in GitHub (should normally not be needed)
testdeploy — in a website-related pull request, takes the website built in GitLab-CI and publishes it under a temporary address (works only after the GitLab-CI build completes and only for selected users)
More tests, specifically for reproducible builds of packages
We also have a nice summary page (https://qubesos.gitlab.io/qubes-g2g-report/) with all the recent jobs states, which is updated daily.
Reproducible builds
Our current short-term goal for reproducible builds in Qubes OS is to integrate what is realistically possible in Debian. Specifically, we aim to get our Debian templates to the state in which packages can be installed only when configured rebuilders confirm they really came from the source code we publish. (A “rebuilder” is a program that takes a binary and its purported source code as inputs, along with any applicable metadata, and attempts to build an identical binary from the source code. The goal is to check whether the binary was really compiled from its claimed source code.) For this task, we focus specifically on Debian Bullseye (testing version), which includes most of the reproducibility work done by the Debian community.
To start, we had to bring our packages to the state in which they could be built reproducibly. GitLab-CI massively helped here with identifying issues. One of the CI jobs we added is running the reprotest tool on our packages. This tool builds the package twice in two different build environments and compares the results. If there are any differences, the job automatically runs the diffoscope tool, which provides a detailed, in-depth report.
Now, all our packages for Debian Bullseye are reproducible, and we are actively monitoring it with our CI. This is designated by the “reproducible” badge on the CI summary page (https://qubesos.gitlab.io/qubes-g2g-report/).
The next step on the journey is to enable an arbitrary user or organization to challenge the official packages — to take the sources, together with the buildinfo file describing how it was built and to verify that they result in the same binary package. Theoretically, there is a debrebuild.pl noscript in Debian’s devnoscripts package that does exactly this. We started by adding missing parts to it, but it turned out there were several limitations making it unsuitable for our use case. The primary limitation is support for only one source repository. (We need two: original Debian and Qubes.) Since debrebuild.pl, as the name suggests, is written in Perl, it isn’t very friendly for contributing to. For these reasons, we decided to write our own similar tool in Python. While this was a bit of work, the end result (https://github.com/fepitre/debrebuild) is very nice: about 900 lines of code, with proper structure using classes. We are in the process of upstreaming this code back to Debian, potentially to replace the original debrebuild.pl tool.