On affected systems, an attacker might be able to infer the contents
of arbitrary host memory, including memory assigned to other guests.
Credits
--------
See the original Xen Security Advisory.
References
-----------
[1] https://www.qubes-os.org/doc/testing/
[2] https://www.qubes-os.org/doc/how-to-update/
[3] https://xenbits.xen.org/xsa/advisory-398.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
of arbitrary host memory, including memory assigned to other guests.
Credits
--------
See the original Xen Security Advisory.
References
-----------
[1] https://www.qubes-os.org/doc/testing/
[2] https://www.qubes-os.org/doc/how-to-update/
[3] https://xenbits.xen.org/xsa/advisory-398.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
Whonix support for Qubes 4.0 extended
https://www.qubes-os.org/news/2022/03/17/whonix-support-for-qubes-4-0-extended/
The Whonix Project has extended its support for Whonix templates on
Qubes 4.0 by an additional month to 2022-04-20. This comes after an
initial 16-day extension when the release of Qubes 4.1 was originally
announced (https://www.qubes-os.org/news/2022/02/04/qubes-4-1-0/).
How does Whonix support work?
While Qubes OS releases are supported for six months (https://www.qubes-os.org/doc/supported-releases/#qubes-os) following each
subsequent major or minor release, Whonix templates have their own
support policy (https://www.qubes-os.org/doc/supported-releases/#note-on-whonix-support) set by the Whonix Project. This policy requires Whonix
template users to stay reasonably close to the cutting edge by upgrading
to new stable releases of Qubes OS and Whonix templates within a month
of their respective releases. To be precise:
One month after a new stable version of Qubes OS is released, Whonix
templates are no longer supported on any older release of Qubes OS.
This means that users who wish to continue using Whonix templates on
Qubes are usually required to upgrade to the latest stable Qubes OS
version within one month of its release (unless the deadline is
extended, as it has been in this case).
One month after new stable versions of Whonix templates are released,
older releases of Whonix templates are no longer supported. This means
that users who wish to continue using Whonix templates on Qubes are
usually required to upgrade to the latest stable Whonix template
versions within one month of their release. (This point doesn’t apply
to the present situation, since this announcement pertains to a new
Qubes OS release, not a new Whonix template release. However, I’m
mentioning it here for the sake of completeness, as it’s part of the
Whonix support policy.)
What does this mean for you?
If you haven’t upgraded from Qubes 4.0 to Qubes 4.1 yet, and you use
Whonix templates, this extension means you have a bit more time to
upgrade before your Whonix templates are no longer supported. You should
aim to upgrade to Qubes 4.1 (or discontinue use of Whonix templates) by
2022-04-20.
If you’re already on Qubes 4.1 or you don’t use Whonix templates, you’re
all set. This announcement doesn’t change anything for you.
https://www.qubes-os.org/news/2022/03/17/whonix-support-for-qubes-4-0-extended/
The Whonix Project has extended its support for Whonix templates on
Qubes 4.0 by an additional month to 2022-04-20. This comes after an
initial 16-day extension when the release of Qubes 4.1 was originally
announced (https://www.qubes-os.org/news/2022/02/04/qubes-4-1-0/).
How does Whonix support work?
While Qubes OS releases are supported for six months (https://www.qubes-os.org/doc/supported-releases/#qubes-os) following each
subsequent major or minor release, Whonix templates have their own
support policy (https://www.qubes-os.org/doc/supported-releases/#note-on-whonix-support) set by the Whonix Project. This policy requires Whonix
template users to stay reasonably close to the cutting edge by upgrading
to new stable releases of Qubes OS and Whonix templates within a month
of their respective releases. To be precise:
One month after a new stable version of Qubes OS is released, Whonix
templates are no longer supported on any older release of Qubes OS.
This means that users who wish to continue using Whonix templates on
Qubes are usually required to upgrade to the latest stable Qubes OS
version within one month of its release (unless the deadline is
extended, as it has been in this case).
One month after new stable versions of Whonix templates are released,
older releases of Whonix templates are no longer supported. This means
that users who wish to continue using Whonix templates on Qubes are
usually required to upgrade to the latest stable Whonix template
versions within one month of their release. (This point doesn’t apply
to the present situation, since this announcement pertains to a new
Qubes OS release, not a new Whonix template release. However, I’m
mentioning it here for the sake of completeness, as it’s part of the
Whonix support policy.)
What does this mean for you?
If you haven’t upgraded from Qubes 4.0 to Qubes 4.1 yet, and you use
Whonix templates, this extension means you have a bit more time to
upgrade before your Whonix templates are no longer supported. You should
aim to upgrade to Qubes 4.1 (or discontinue use of Whonix templates) by
2022-04-20.
If you’re already on Qubes 4.1 or you don’t use Whonix templates, you’re
all set. This announcement doesn’t change anything for you.
Qubes OS configuration survey! (5-10 minutes)
https://www.qubes-os.org/news/2022/03/23/qubes-os-configuration-survey/
We’re working on a simpler, more user-friendly Qubes OS configuration
experience. We invite you all to lend us 5-10 minutes of your time to
participate in this 100% anonymous survey. Your participation will help
us build better GUI tools for system configuration that meet real user
needs. This survey will remain live for 28 days. Thank you!
https://survey.qubes-os.org/index.php?r=survey/index&sid=762843&lang=en
https://www.qubes-os.org/news/2022/03/23/qubes-os-configuration-survey/
We’re working on a simpler, more user-friendly Qubes OS configuration
experience. We invite you all to lend us 5-10 minutes of your time to
participate in this 100% anonymous survey. Your participation will help
us build better GUI tools for system configuration that meet real user
needs. This survey will remain live for 28 days. Thank you!
https://survey.qubes-os.org/index.php?r=survey/index&sid=762843&lang=en
MirageOS Announces Latest Release v4.0, dedicated to Lars Kurth
https://xenproject.org/2022/03/30/mirageos-announces-latest-release-v4-0-dedicated-to-lars-kurth/
MirageOS core maintainers and the Linux Foundation announced the release of MirageOS version 4.0, the latest update since version 3.10 in December, 2020. SAN FRANCISCO, March 29, 2022 — The MirageOS...
https://xenproject.org/2022/03/30/mirageos-announces-latest-release-v4-0-dedicated-to-lars-kurth/
MirageOS core maintainers and the Linux Foundation announced the release of MirageOS version 4.0, the latest update since version 3.10 in December, 2020. SAN FRANCISCO, March 29, 2022 — The MirageOS...
XSAs released on 2022-04-05
https://www.qubes-os.org/news/2022/04/05/xsas-released-on-2022-04-05/
The Xen Project has released one or more Xen Security Advisories (XSAs).
The security of Qubes OS is affected.
Therefore, user action is required.
XSAs that affect the security of Qubes OS (user action required)
The following XSAs do affect the security of Qubes OS:
XSA-399
XSA-400
Please see QSB-079 for the actions users must take in order to
protect themselves, as well as further details about these XSAs:
https://www.qubes-os.org/news/2022/04/05/qsb-079/
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
XSA-397 (denial of service only)
Related links
Xen XSA list: https://xenbits.xen.org/xsa/
Qubes XSA tracker: https://www.qubes-os.org/security/xsa/
Qubes security pack (qubes-secpack): https://www.qubes-os.org/security/pack/
Qubes security bulletins (QSBs): https://www.qubes-os.org/security/qsb/
https://www.qubes-os.org/news/2022/04/05/xsas-released-on-2022-04-05/
The Xen Project has released one or more Xen Security Advisories (XSAs).
The security of Qubes OS is affected.
Therefore, user action is required.
XSAs that affect the security of Qubes OS (user action required)
The following XSAs do affect the security of Qubes OS:
XSA-399
XSA-400
Please see QSB-079 for the actions users must take in order to
protect themselves, as well as further details about these XSAs:
https://www.qubes-os.org/news/2022/04/05/qsb-079/
XSAs that do not affect the security of Qubes OS (no user action required)
The following XSAs do not affect the security of Qubes OS, and no user action is necessary:
XSA-397 (denial of service only)
Related links
Xen XSA list: https://xenbits.xen.org/xsa/
Qubes XSA tracker: https://www.qubes-os.org/security/xsa/
Qubes security pack (qubes-secpack): https://www.qubes-os.org/security/pack/
Qubes security bulletins (QSBs): https://www.qubes-os.org/security/qsb/
QSB-079: Two IOMMU-related Xen issues (XSA-399, XSA-400)
https://www.qubes-os.org/news/2022/04/05/qsb-079/
We have just published Qubes Security Bulletin (QSB) 079:
Two IOMMU-related Xen issues (XSA-399, XSA-400).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-079 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-079-2022.txt
In addition, you may wish to:
Get the qubes-secpack: https://www.qubes-os.org/security/pack/
View all past QSBs: https://www.qubes-os.org/security/qsb/
View the XSA Tracker: https://www.qubes-os.org/security/xsa/
---===[ Qubes Security Bulletin 079 ]===---
2022-04-05
Two IOMMU-related Xen issues (XSA-399, XSA-400)
User action required
---------------------
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0, in dom0:
- Xen packages, version 4.8.5-39
For Qubes 4.1, in dom0:
- Xen packages, version 4.14.4-4
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community. [1] Once available, the packages are to be installed
via the Qubes Update tool or its command-line equivalents. [2]
Dom0 must be restarted afterward in order for the updates to take
effect.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Xen binaries.
Summary
--------
The following security advisories were published on 2022-04-05:
XSA-399 [3] "race in VT-d domain ID cleanup":
| Xen domain IDs are up to 15 bits wide. VT-d hardware may allow for only
| less than 15 bits to hold a domain ID associating a physical device with
| a particular domain. Therefore internally Xen domain IDs are mapped to
| the smaller value range. The cleaning up of the housekeeping structures
| has a race, allowing for VT-d domain IDs to be leaked and flushes to be
| bypassed.
XSA-400 [4] "IOMMU: RMRR (VT-d) and unity map (AMD-Vi) handling
issues":
| Certain PCI devices in a system might be assigned Reserved Memory
| Regions (specified via Reserved Memory Region Reporting, "RMRR") for
| Intel VT-d or Unity Mapping ranges for AMD-Vi. These are typically used
| for platform tasks such as legacy USB emulation.
|
| Since the precise purpose of these regions is unknown, once a device
| associated with such a region is active, the mappings of these regions
| need to remain continuouly accessible by the device. This requirement
| has been violated.
|
| Subsequent DMA or interrupts from the device may have unpredictable
| behaviour, ranging from IOMMU faults to memory corruption.
Impact
-------
The precise impact of XSA-399 and XSA-400 is system-specific but
would typically be a denial of service (DoS) affecting the entire
host. Privilege escalation and information leaks cannot be ruled out.
XSA-399 affects only Qubes OS 4.1. Qubes OS 4.0 is not affected due to
the version of Xen it uses. This issue affects only qubes with assigned
PCI devices, which are sys-net and sys-usb in the default Qubes OS
configuration.
XSA-400 affects both Qubes OS 4.0 and 4.1. It affects only qubes with
assigned PCI devices that have an associated RMRR or unity map. This
usually applies to USB controllers, which are assigned to the sys-usb
qube in the default Qubes OS configuration.
Credits
--------
See the original Xen Security Advisory.
References
-----------
[1] https://www.qubes-os.org/doc/testing/
[2] https://www.qubes-os.org/doc/how-to-update/
[3] https://xenbits.xen.org/xsa/advisory-399.html
[4] https://xenbits.xen.org/xsa/advisory-400.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
https://www.qubes-os.org/news/2022/04/05/qsb-079/
We have just published Qubes Security Bulletin (QSB) 079:
Two IOMMU-related Xen issues (XSA-399, XSA-400).
The text of this QSB is reproduced below. This QSB and its accompanying
signatures will always be available in the Qubes Security Pack (qubes-secpack).
View QSB-079 in the qubes-secpack:
https://github.com/QubesOS/qubes-secpack/blob/master/QSBs/qsb-079-2022.txt
In addition, you may wish to:
Get the qubes-secpack: https://www.qubes-os.org/security/pack/
View all past QSBs: https://www.qubes-os.org/security/qsb/
View the XSA Tracker: https://www.qubes-os.org/security/xsa/
---===[ Qubes Security Bulletin 079 ]===---
2022-04-05
Two IOMMU-related Xen issues (XSA-399, XSA-400)
User action required
---------------------
Users must install the following specific packages in order to address
the issues discussed in this bulletin:
For Qubes 4.0, in dom0:
- Xen packages, version 4.8.5-39
For Qubes 4.1, in dom0:
- Xen packages, version 4.14.4-4
These packages will migrate from the security-testing repository to the
current (stable) repository over the next two weeks after being tested
by the community. [1] Once available, the packages are to be installed
via the Qubes Update tool or its command-line equivalents. [2]
Dom0 must be restarted afterward in order for the updates to take
effect.
If you use Anti Evil Maid, you will need to reseal your secret
passphrase to new PCR values, as PCR18+19 will change due to the new
Xen binaries.
Summary
--------
The following security advisories were published on 2022-04-05:
XSA-399 [3] "race in VT-d domain ID cleanup":
| Xen domain IDs are up to 15 bits wide. VT-d hardware may allow for only
| less than 15 bits to hold a domain ID associating a physical device with
| a particular domain. Therefore internally Xen domain IDs are mapped to
| the smaller value range. The cleaning up of the housekeeping structures
| has a race, allowing for VT-d domain IDs to be leaked and flushes to be
| bypassed.
XSA-400 [4] "IOMMU: RMRR (VT-d) and unity map (AMD-Vi) handling
issues":
| Certain PCI devices in a system might be assigned Reserved Memory
| Regions (specified via Reserved Memory Region Reporting, "RMRR") for
| Intel VT-d or Unity Mapping ranges for AMD-Vi. These are typically used
| for platform tasks such as legacy USB emulation.
|
| Since the precise purpose of these regions is unknown, once a device
| associated with such a region is active, the mappings of these regions
| need to remain continuouly accessible by the device. This requirement
| has been violated.
|
| Subsequent DMA or interrupts from the device may have unpredictable
| behaviour, ranging from IOMMU faults to memory corruption.
Impact
-------
The precise impact of XSA-399 and XSA-400 is system-specific but
would typically be a denial of service (DoS) affecting the entire
host. Privilege escalation and information leaks cannot be ruled out.
XSA-399 affects only Qubes OS 4.1. Qubes OS 4.0 is not affected due to
the version of Xen it uses. This issue affects only qubes with assigned
PCI devices, which are sys-net and sys-usb in the default Qubes OS
configuration.
XSA-400 affects both Qubes OS 4.0 and 4.1. It affects only qubes with
assigned PCI devices that have an associated RMRR or unity map. This
usually applies to USB controllers, which are assigned to the sys-usb
qube in the default Qubes OS configuration.
Credits
--------
See the original Xen Security Advisory.
References
-----------
[1] https://www.qubes-os.org/doc/testing/
[2] https://www.qubes-os.org/doc/how-to-update/
[3] https://xenbits.xen.org/xsa/advisory-399.html
[4] https://xenbits.xen.org/xsa/advisory-400.html
--
The Qubes Security Team
https://www.qubes-os.org/security/
⚠️ This channel is updated ASAP after devs make an announcement to the project.
Help?
English Group: t.me/joinchat/B8FHpkEToMfgdREGV7wzRQ
German Group: @QubesOS_user_de
Help?
English Group: t.me/joinchat/B8FHpkEToMfgdREGV7wzRQ
German Group: @QubesOS_user_de
Telegram
QubesOS Chat
(Community Support)
Alternative community instead of online forum and discord.
@QubesOS
>No spam/ads/NSFW/shit posts/spam bots
>Respect others
>Stay on topic
>Be transparent
If you need help ask!
Rules are enforced with ban or restrictions.
Alternative community instead of online forum and discord.
@QubesOS
>No spam/ads/NSFW/shit posts/spam bots
>Respect others
>Stay on topic
>Be transparent
If you need help ask!
Rules are enforced with ban or restrictions.
Windows integration work in Qubes 4.1 by the tabit-pro team
https://www.qubes-os.org/news/2022/04/10/windows-integration-by-tabit-pro/
Editor’s note: This is a guest article by Ivan Kardykov from
tabit-pro (https://tabit.pro/). We’ve invited Ivan to explain the work
the tabit-pro team contributed to Qubes 4.1. Welcome, Ivan!
In this article, I’ll briefly describe the code contributions we made to
the latest Qubes 4.1 release, most of which focus on improving Windows
integration.
OEM activation support
When Windows comes preinstalled on a computer, license activation is
based on a certificate embedded in the hardware. Technically, this uses
one of the ACPI tables called “SLIC,” which is readable by the host OS.
This option is not available in Qubes OS, since each qube is a Xen
virtual machine (VM) that has no physical hardware of its own. However,
a small change in Xen allowed us to copy the necessary data onto the
appropriate memory partition of the VM (thanks to OpenXT for the working
patch). This can be done simply by extending the VM configuration with
the SLIC data via the libvirt template
extension (https://github.com/QubesOS/qubes-issues/issues/5279#issuecomment-525947408).
This fix has been included in stable packages for a long time. In fact,
it is also available for Qubes 4.0 users.
Audio support
Audio virtualization in Qubes OS is based on communication between
PulseAudio services running in each VM, including dom0 (described in
more detail here (https://www.qubes-os.org/doc/audio-virtualization/)). Unfortunately, this
method is problematic in the case of Windows VMs due to the lack of
PulseAudio support in Windows, and attempting to support it
independently seems like too time-consuming of a task (see
#2624 (https://github.com/QubesOS/qubes-issues/issues/2624)).
An alternative is to use QEMU, which allows for the emulation of
different audio devices and docking with PulseAudio. In our case, the
main obstacle for this method is the complexity of the connection to
QEMU, which is isolated by a stubdomain. We developed a patch that
allows for building and starting the necessary components for PulseAudio
in a minimal environment with vchan support. We also worked out a
separate version for building the stubdomain image with extended
functionality (the xen-hvm-stubdom-linux-full package). This mode is
activated in HVMs by setting the audio-model feature (using
qvm-features) and specifying the type of audio device, for example,
ich6. (Device variants are described in the QEMU documentation.)
USB support
We used a similar approach to improve support for USB devices. In the
extended stubdomain, we proposed including qrexec and USB-proxy services
and including libusb features in QEMU (see
qubes-app-linux-usb-proxy (https://github.com/QubesOS/qubes-app-linux-usb-proxy)).
As a result, we didn’t even need to increase the available memory in
order to achieve stable operation of the extended stubdomain, although
it may be a problem when using some devices (e.g., webcams). There was a
question of which controller type QEMU should emulate. After
experimenting with different devices, we settled on the NEC XHCI, in
part due to the availability of Windows 7 drivers. The activation of
this emulation mode in HVMs is done by setting the stubdom-qrexec
property (using qvm-features). Details of user testing can be found on
the Qubes
Forum (https://forum.qubes-os.org/t/windows-usb-integration-with-r4-1/5001).
Not only Windows
The advantage of this approach is that there is no need to install guest
tools for attaching audio and USB devices, which allows the emulation to
be used not only with Windows, but with any OS (e.g., Linux live images,
Android x86, and probably ReactOS). At the same time, there are also
disadvantages. In particular, the additional workload can slow down the
VM and affect the sound quality. (Even at minimum workload, you may
notice crackling.)
Qubes Windows Tools
https://www.qubes-os.org/news/2022/04/10/windows-integration-by-tabit-pro/
Editor’s note: This is a guest article by Ivan Kardykov from
tabit-pro (https://tabit.pro/). We’ve invited Ivan to explain the work
the tabit-pro team contributed to Qubes 4.1. Welcome, Ivan!
In this article, I’ll briefly describe the code contributions we made to
the latest Qubes 4.1 release, most of which focus on improving Windows
integration.
OEM activation support
When Windows comes preinstalled on a computer, license activation is
based on a certificate embedded in the hardware. Technically, this uses
one of the ACPI tables called “SLIC,” which is readable by the host OS.
This option is not available in Qubes OS, since each qube is a Xen
virtual machine (VM) that has no physical hardware of its own. However,
a small change in Xen allowed us to copy the necessary data onto the
appropriate memory partition of the VM (thanks to OpenXT for the working
patch). This can be done simply by extending the VM configuration with
the SLIC data via the libvirt template
extension (https://github.com/QubesOS/qubes-issues/issues/5279#issuecomment-525947408).
This fix has been included in stable packages for a long time. In fact,
it is also available for Qubes 4.0 users.
Audio support
Audio virtualization in Qubes OS is based on communication between
PulseAudio services running in each VM, including dom0 (described in
more detail here (https://www.qubes-os.org/doc/audio-virtualization/)). Unfortunately, this
method is problematic in the case of Windows VMs due to the lack of
PulseAudio support in Windows, and attempting to support it
independently seems like too time-consuming of a task (see
#2624 (https://github.com/QubesOS/qubes-issues/issues/2624)).
An alternative is to use QEMU, which allows for the emulation of
different audio devices and docking with PulseAudio. In our case, the
main obstacle for this method is the complexity of the connection to
QEMU, which is isolated by a stubdomain. We developed a patch that
allows for building and starting the necessary components for PulseAudio
in a minimal environment with vchan support. We also worked out a
separate version for building the stubdomain image with extended
functionality (the xen-hvm-stubdom-linux-full package). This mode is
activated in HVMs by setting the audio-model feature (using
qvm-features) and specifying the type of audio device, for example,
ich6. (Device variants are described in the QEMU documentation.)
USB support
We used a similar approach to improve support for USB devices. In the
extended stubdomain, we proposed including qrexec and USB-proxy services
and including libusb features in QEMU (see
qubes-app-linux-usb-proxy (https://github.com/QubesOS/qubes-app-linux-usb-proxy)).
As a result, we didn’t even need to increase the available memory in
order to achieve stable operation of the extended stubdomain, although
it may be a problem when using some devices (e.g., webcams). There was a
question of which controller type QEMU should emulate. After
experimenting with different devices, we settled on the NEC XHCI, in
part due to the availability of Windows 7 drivers. The activation of
this emulation mode in HVMs is done by setting the stubdom-qrexec
property (using qvm-features). Details of user testing can be found on
the Qubes
Forum (https://forum.qubes-os.org/t/windows-usb-integration-with-r4-1/5001).
Not only Windows
The advantage of this approach is that there is no need to install guest
tools for attaching audio and USB devices, which allows the emulation to
be used not only with Windows, but with any OS (e.g., Linux live images,
Android x86, and probably ReactOS). At the same time, there are also
disadvantages. In particular, the additional workload can slow down the
VM and affect the sound quality. (Even at minimum workload, you may
notice crackling.)
Qubes Windows Tools
A relatively long time ago, we proposed building all the components of
Qubes Windows Tools (QWT) in a Linux environment using MinGW and Wine,
which allows us to use our existing CI tools and simplify the
maintenance of changes. We also worked a lot on improving the stability
of all the components, in particular, eliminating the causes of freezes
and performance degradation. In recent weeks, all of our proposed
changes have been approved and merged, and the
cross-build (https://github.com/QubesOS/qubes-windows-tools-cross)
project has been added as a Qubes OS component. I’m sure everything will
be available in the stable repositories soon.
Conclusion
The result of our work is full integration of Windows 7, 10, and 11 in
Qubes OS and significant improvements in usability for even
inexperienced users.
Thanks to all members of the tabit-pro team, especially Dmitriy Fedorov
(@easydozen (https://github.com/easydozen)). Many thanks to Marek
Marczykowski-Górecki for his assistance and the Qubes dev team as a
whole for their daily work. Special thanks to the Qubes community,
especially for their kind words of support.
Ivan Kardykov
tabit.pro (https://tabit.pro/)
Qubes Windows Tools (QWT) in a Linux environment using MinGW and Wine,
which allows us to use our existing CI tools and simplify the
maintenance of changes. We also worked a lot on improving the stability
of all the components, in particular, eliminating the causes of freezes
and performance degradation. In recent weeks, all of our proposed
changes have been approved and merged, and the
cross-build (https://github.com/QubesOS/qubes-windows-tools-cross)
project has been added as a Qubes OS component. I’m sure everything will
be available in the stable repositories soon.
Conclusion
The result of our work is full integration of Windows 7, 10, and 11 in
Qubes OS and significant improvements in usability for even
inexperienced users.
Thanks to all members of the tabit-pro team, especially Dmitriy Fedorov
(@easydozen (https://github.com/easydozen)). Many thanks to Marek
Marczykowski-Górecki for his assistance and the Qubes dev team as a
whole for their daily work. Special thanks to the Qubes community,
especially for their kind words of support.
Ivan Kardykov
tabit.pro (https://tabit.pro/)
🔥3
Whonix support for Qubes 4.0 has ended
https://www.qubes-os.org/news/2022/04/20/whonix-support-for-qubes-4-0-has-ended/
As a final reminder following our previous (https://www.qubes-os.org/news/2022/02/04/qubes-4-1-0/#support-for-older-releases) announcements (https://www.qubes-os.org/news/2022/03/17/whonix-support-for-qubes-4-0-extended/), Whonix
support for Qubes 4.0 ends today, 2022-04-20.
How does Whonix support work?
While Qubes OS releases are supported for six months (https://www.qubes-os.org/doc/supported-releases/#qubes-os) following each
subsequent major or minor release, Whonix templates have their own
support policy (https://www.qubes-os.org/doc/supported-releases/#note-on-whonix-support) set by the Whonix Project. This policy requires Whonix
template users to stay reasonably close to the cutting edge by upgrading
to new stable releases of Qubes OS and Whonix templates within a month
of their respective releases. To be precise:
One month after a new stable version of Qubes OS is released, Whonix
templates are no longer supported on any older release of Qubes OS.
This means that users who wish to continue using Whonix templates on
Qubes are usually required to upgrade to the latest stable Qubes OS
version within one month of its release (unless the deadline is
extended, as it has been in this case).
One month after new stable versions of Whonix templates are released,
older releases of Whonix templates are no longer supported. This means
that users who wish to continue using Whonix templates on Qubes are
usually required to upgrade to the latest stable Whonix template
versions within one month of their release. (This point doesn’t apply
to the present situation, since this announcement pertains to a new
Qubes OS release, not a new Whonix template release. However, I’m
mentioning it here for the sake of completeness, as it’s part of the
Whonix support policy.)
What does this mean for you?
If you’re currently using Whonix on Qubes 4.0, you should immediately
either upgrade to Qubes 4.1 (https://www.qubes-os.org/doc/upgrade/4.1/) or discontinue the use of Whonix.
If you’re already on Qubes 4.1, or you don’t use Whonix templates,
then you’re all set. This announcement doesn’t change anything for you.
https://www.qubes-os.org/news/2022/04/20/whonix-support-for-qubes-4-0-has-ended/
As a final reminder following our previous (https://www.qubes-os.org/news/2022/02/04/qubes-4-1-0/#support-for-older-releases) announcements (https://www.qubes-os.org/news/2022/03/17/whonix-support-for-qubes-4-0-extended/), Whonix
support for Qubes 4.0 ends today, 2022-04-20.
How does Whonix support work?
While Qubes OS releases are supported for six months (https://www.qubes-os.org/doc/supported-releases/#qubes-os) following each
subsequent major or minor release, Whonix templates have their own
support policy (https://www.qubes-os.org/doc/supported-releases/#note-on-whonix-support) set by the Whonix Project. This policy requires Whonix
template users to stay reasonably close to the cutting edge by upgrading
to new stable releases of Qubes OS and Whonix templates within a month
of their respective releases. To be precise:
One month after a new stable version of Qubes OS is released, Whonix
templates are no longer supported on any older release of Qubes OS.
This means that users who wish to continue using Whonix templates on
Qubes are usually required to upgrade to the latest stable Qubes OS
version within one month of its release (unless the deadline is
extended, as it has been in this case).
One month after new stable versions of Whonix templates are released,
older releases of Whonix templates are no longer supported. This means
that users who wish to continue using Whonix templates on Qubes are
usually required to upgrade to the latest stable Whonix template
versions within one month of their release. (This point doesn’t apply
to the present situation, since this announcement pertains to a new
Qubes OS release, not a new Whonix template release. However, I’m
mentioning it here for the sake of completeness, as it’s part of the
Whonix support policy.)
What does this mean for you?
If you’re currently using Whonix on Qubes 4.0, you should immediately
either upgrade to Qubes 4.1 (https://www.qubes-os.org/doc/upgrade/4.1/) or discontinue the use of Whonix.
If you’re already on Qubes 4.1, or you don’t use Whonix templates,
then you’re all set. This announcement doesn’t change anything for you.
Automated OS testing on physical laptops
https://www.qubes-os.org/news/2022/05/05/automated-os-testing-on-physical-laptops/
Our journey towards automating OS tests on physical laptops started a few years
ago with the idea of using Intel AMT to drive tests on physical machines. To
start, I got an initial
implementation (https://github.com/os-autoinst/os-autoinst/pull/983) working.
In particular, VNC for input/output and power control worked. I tried to get a
virtual CD working, but it turned out to be quite unstable. Worse — and more
importantly — it was really just a CD, not a CD/DVD, which meant that the
protocol couldn’t handle images larger than 2 GB. Some time later I abandoned
this approach, for two related reasons:
Many machines that we want Qubes OS to support intentionally do not have
Intel AMT.
The single AMT-enabled machine that I had been using to develop this feature
broke.
If anyone would like to resume this work, this
page (https://web.archive.org/web/20200926070850/senseless.info/amt.html)
includes a lot of useful info about Intel AMT on Linux.
Recently, I came back to the project with a new approach: to capture video from
HDMI output and use an emulated USB keyboard and mouse for input. Then, I added
power control to the mix, combined everything on a Raspberry Pi, and got a
working prototype (https://github.com/os-autoinst/os-autoinst/pull/1741) of an
openQA worker that runs the tests on a physical machine, instead of a virtual
one.
The whole setup includes several devices:
One “central” Raspberry Pi that controls a power strip and serves boot files.
One Raspberry Pi per laptop that runs an openQA worker for that laptop. It
emulates a USB device for that laptop and captures HDMI output from it.
All these elements are detailed below.
Base system
The goal was to run an openQA worker on a Raspberry Pi 4. Why a Raspberry Pi
(RPi)?
Their USB controllers can play the role of a device, not just that of USB
host.
They’re powerful enough to run the video processing required by openQA.
They’re (mostly) readily available and relatively cheap.
As a base system, I chose OpenSUSE, because that’s openQA’s native
distribution. Getting OpenSUSE to work on an RPi was rather
straightforward (https://en.opensuse.org/HCL:Raspberry_Pi4), but the choice did
lead to a few issues discussed later in this article.
Power control
Power control was the first stage of this project. I thought it looked like the
simplest part.
To reliably run unattended tests, I needed a way to interrupt a test when it
went into some unrecoverable state (kernel panic, hard hang, etc.). With AMT, I
had a built-in API for that, but now I needed something else. I chose a power
strip that was remotely controlled via USB. Then, I removed the batteries from
the laptops connected to the setup. This gave me a very reliable way to
interrupt whatever was running on the machines by simply powering them down.
But it turned out that powering them back on may not be that simple.
In the current setup, there are several laptops, each of them slightly
different, and each (sic!) requiring a slightly different approach to power
management. Here are some things I tried and that worked on some machines:
Setting the BIOS to automatically power on the machine when a power supply
was connected. This is the simplest method. Sadly, only one of the machines
supported it.
Sending a Wake-On-Lan packet. Here, reliability depends on the device. For
some, it just works, while others require enabling it in the network card
(with the ethtool -s eth0 wol g command), and some lose the setting either
on system startup or on disconnecting the power…
When all else fails, one can just press the physical power button. Of course
it would be too much work to do it manually, so I attached a servo motor in
the exact spot where the power button is. Then, I drove that servo motor
from an RPi.
https://www.qubes-os.org/news/2022/05/05/automated-os-testing-on-physical-laptops/
Our journey towards automating OS tests on physical laptops started a few years
ago with the idea of using Intel AMT to drive tests on physical machines. To
start, I got an initial
implementation (https://github.com/os-autoinst/os-autoinst/pull/983) working.
In particular, VNC for input/output and power control worked. I tried to get a
virtual CD working, but it turned out to be quite unstable. Worse — and more
importantly — it was really just a CD, not a CD/DVD, which meant that the
protocol couldn’t handle images larger than 2 GB. Some time later I abandoned
this approach, for two related reasons:
Many machines that we want Qubes OS to support intentionally do not have
Intel AMT.
The single AMT-enabled machine that I had been using to develop this feature
broke.
If anyone would like to resume this work, this
page (https://web.archive.org/web/20200926070850/senseless.info/amt.html)
includes a lot of useful info about Intel AMT on Linux.
Recently, I came back to the project with a new approach: to capture video from
HDMI output and use an emulated USB keyboard and mouse for input. Then, I added
power control to the mix, combined everything on a Raspberry Pi, and got a
working prototype (https://github.com/os-autoinst/os-autoinst/pull/1741) of an
openQA worker that runs the tests on a physical machine, instead of a virtual
one.
The whole setup includes several devices:
One “central” Raspberry Pi that controls a power strip and serves boot files.
One Raspberry Pi per laptop that runs an openQA worker for that laptop. It
emulates a USB device for that laptop and captures HDMI output from it.
All these elements are detailed below.
Base system
The goal was to run an openQA worker on a Raspberry Pi 4. Why a Raspberry Pi
(RPi)?
Their USB controllers can play the role of a device, not just that of USB
host.
They’re powerful enough to run the video processing required by openQA.
They’re (mostly) readily available and relatively cheap.
As a base system, I chose OpenSUSE, because that’s openQA’s native
distribution. Getting OpenSUSE to work on an RPi was rather
straightforward (https://en.opensuse.org/HCL:Raspberry_Pi4), but the choice did
lead to a few issues discussed later in this article.
Power control
Power control was the first stage of this project. I thought it looked like the
simplest part.
To reliably run unattended tests, I needed a way to interrupt a test when it
went into some unrecoverable state (kernel panic, hard hang, etc.). With AMT, I
had a built-in API for that, but now I needed something else. I chose a power
strip that was remotely controlled via USB. Then, I removed the batteries from
the laptops connected to the setup. This gave me a very reliable way to
interrupt whatever was running on the machines by simply powering them down.
But it turned out that powering them back on may not be that simple.
In the current setup, there are several laptops, each of them slightly
different, and each (sic!) requiring a slightly different approach to power
management. Here are some things I tried and that worked on some machines:
Setting the BIOS to automatically power on the machine when a power supply
was connected. This is the simplest method. Sadly, only one of the machines
supported it.
Sending a Wake-On-Lan packet. Here, reliability depends on the device. For
some, it just works, while others require enabling it in the network card
(with the ethtool -s eth0 wol g command), and some lose the setting either
on system startup or on disconnecting the power…
When all else fails, one can just press the physical power button. Of course
it would be too much work to do it manually, so I attached a servo motor in
the exact spot where the power button is. Then, I drove that servo motor
from an RPi.
👍1
System startup
After achieving control over system power, the next step was taking control of
which operating system starts there. I considered two options:
A USB boot drive, emulated from an RPi
Network boot
The first option turned out to be problematic when combined with emulated USB
input devices (see below), at least on some laptops. While a single USB device
can have multiple interfaces (basically being sub-devices), many types of
system firmware do not like to boot from such devices. When I exposed a USB
device that has both a storage interface and a HID interface (keyboard/mouse),
the system didn’t consider it a bootable device. One solution would be to use
two separate devices, but that would require yet another RPi (or something
similar), since most (all?) such boards support emulating only a single device.
Another way around it could be emulating a USB hub and getting two virtual
devices this way, but Linux does not support USB hub emulation. Since I had an
alternative, I didn’t explore this option any further. On systems that are fine
with a single multi-function USB device, I can use that. On others, I use
network boot.
The second option turned out not to be that straightforward either. First of
all, not all systems support booting from the network to begin with. To solve
this problem, I got a USB stick and put iPXE (https://ipxe.org/) on it. Then, I
configured the system to boot from that USB stick. I couldn’t use Grub here to
gain network boot, because Grub supports only network devices via the system
firmware (BIOS/EFI) support, and this support is missing on systems not capable
of network booting. iPXE, on the other hand, supports a wide range of network
devices on its own, and also allows simple noscripting, like booting different
systems depending on various settings. Unfortunately, it cannot boot Xen via
the multiboot2 protocol (required to boot with full EFI support), it can only
do multiboot1. So, I did need Grub. Luckily, iPXE does register its drivers as
appropriate EFI services, so when I load Grub from iPXE, it can talk to the
network.
I prepared a Grub configuration that can boot different systems on different
laptops depending on a separate configuration file (loaded via the load_env
Grub command) and a tool to conveniently switch between them. This got me a
nice menu:
$ testbed-control 2 help
Selected target: 2
Available commands:
- reset - hard reset the target
- poweron - power on the target
- poweroff - (hard) power off the target
- wake - wake up the system (either wake-on-lan, or button press)
- rescue - switch next boot to rescue system (doesn't load anything from the disk)
- fallback - switch next boot to fallback system (loads /boot/efi/EFI/qubes/grub-fallback.cfg)
- normal - switch next boot to normal system
- custom - switch next boot to custom grub config (/srv/tftp/test2/grub.cfg)
The first four commands are about power control (see above), and the rest are
about choosing what to boot. The normal command simply starts the system
installed on the local disk, while rescue allows booting an initramfs-only
system to diagnose why the normal system doesn’t work. The custom option
allows, in practice, starting an arbitrary kernel (not necessarily from the
disk). That option is especially useful for debugging Linux and Xen issues,
including doing automatic bisection, although it requires a bit more in terms
of glue noscripts (but that’s a topic for another article).
Surprisingly, I had one case where booting the local system turned out to be
tricky. When the bootloader is loaded from the network, that particular UEFI
does not register services to access the local disk. As it turns out, Grub does
not support NVMe drives directly; it supports them only via UEFI services. I
could have switched to another disk, or to booting via USB, but neither of
those options felt appealing. I wanted to run tests on NVMe drives too, and
After achieving control over system power, the next step was taking control of
which operating system starts there. I considered two options:
A USB boot drive, emulated from an RPi
Network boot
The first option turned out to be problematic when combined with emulated USB
input devices (see below), at least on some laptops. While a single USB device
can have multiple interfaces (basically being sub-devices), many types of
system firmware do not like to boot from such devices. When I exposed a USB
device that has both a storage interface and a HID interface (keyboard/mouse),
the system didn’t consider it a bootable device. One solution would be to use
two separate devices, but that would require yet another RPi (or something
similar), since most (all?) such boards support emulating only a single device.
Another way around it could be emulating a USB hub and getting two virtual
devices this way, but Linux does not support USB hub emulation. Since I had an
alternative, I didn’t explore this option any further. On systems that are fine
with a single multi-function USB device, I can use that. On others, I use
network boot.
The second option turned out not to be that straightforward either. First of
all, not all systems support booting from the network to begin with. To solve
this problem, I got a USB stick and put iPXE (https://ipxe.org/) on it. Then, I
configured the system to boot from that USB stick. I couldn’t use Grub here to
gain network boot, because Grub supports only network devices via the system
firmware (BIOS/EFI) support, and this support is missing on systems not capable
of network booting. iPXE, on the other hand, supports a wide range of network
devices on its own, and also allows simple noscripting, like booting different
systems depending on various settings. Unfortunately, it cannot boot Xen via
the multiboot2 protocol (required to boot with full EFI support), it can only
do multiboot1. So, I did need Grub. Luckily, iPXE does register its drivers as
appropriate EFI services, so when I load Grub from iPXE, it can talk to the
network.
I prepared a Grub configuration that can boot different systems on different
laptops depending on a separate configuration file (loaded via the load_env
Grub command) and a tool to conveniently switch between them. This got me a
nice menu:
$ testbed-control 2 help
Selected target: 2
Available commands:
- reset - hard reset the target
- poweron - power on the target
- poweroff - (hard) power off the target
- wake - wake up the system (either wake-on-lan, or button press)
- rescue - switch next boot to rescue system (doesn't load anything from the disk)
- fallback - switch next boot to fallback system (loads /boot/efi/EFI/qubes/grub-fallback.cfg)
- normal - switch next boot to normal system
- custom - switch next boot to custom grub config (/srv/tftp/test2/grub.cfg)
The first four commands are about power control (see above), and the rest are
about choosing what to boot. The normal command simply starts the system
installed on the local disk, while rescue allows booting an initramfs-only
system to diagnose why the normal system doesn’t work. The custom option
allows, in practice, starting an arbitrary kernel (not necessarily from the
disk). That option is especially useful for debugging Linux and Xen issues,
including doing automatic bisection, although it requires a bit more in terms
of glue noscripts (but that’s a topic for another article).
Surprisingly, I had one case where booting the local system turned out to be
tricky. When the bootloader is loaded from the network, that particular UEFI
does not register services to access the local disk. As it turns out, Grub does
not support NVMe drives directly; it supports them only via UEFI services. I
could have switched to another disk, or to booting via USB, but neither of
those options felt appealing. I wanted to run tests on NVMe drives too, and
while USB booting works, it is a bit fragile, because one needs to be careful
not to overwrite that boot drive (especially when testing system
installations). So, I developed a workaround: setting a BootNext EFI variable
(selecting the alternative boot option for just the next startup) and
rebooting. Unfortunately, Grub itself does not have a function to set EFI
variables (it can only read them), but building Linux + minimal initrd with
relevant tools is rather easy. By the way, if I were starting Linux anyway, I
could simply kexec the target kernel from the NVMe disk using Linux’s drivers,
but I wanted the actual startup to remain closer to the “normal” startup,
including respecting the relevant Grub configuration.
There was one final problem to solve. When installing Qubes OS, it will set
itself as the default boot target. This means that all of the above boot
options will be overridden by the installer. To solve this issue, I passed a
kickstart file to the installer that restores the original boot order at the
very last step (%post noscript).
To summarize, I now had:
A way to load Grub2 on each test system (either via PXE or via iPXE loaded
from a USB stick)
A way to conveniently control which OS Grub2 will start
A way to load a local kernel even if Grub2 does not see the disk
not to overwrite that boot drive (especially when testing system
installations). So, I developed a workaround: setting a BootNext EFI variable
(selecting the alternative boot option for just the next startup) and
rebooting. Unfortunately, Grub itself does not have a function to set EFI
variables (it can only read them), but building Linux + minimal initrd with
relevant tools is rather easy. By the way, if I were starting Linux anyway, I
could simply kexec the target kernel from the NVMe disk using Linux’s drivers,
but I wanted the actual startup to remain closer to the “normal” startup,
including respecting the relevant Grub configuration.
There was one final problem to solve. When installing Qubes OS, it will set
itself as the default boot target. This means that all of the above boot
options will be overridden by the installer. To solve this issue, I passed a
kickstart file to the installer that restores the original boot order at the
very last step (%post noscript).
To summarize, I now had:
A way to load Grub2 on each test system (either via PXE or via iPXE loaded
from a USB stick)
A way to conveniently control which OS Grub2 will start
A way to load a local kernel even if Grub2 does not see the disk
Video capture
I started experimenting with HDMI-over-IP extenders. Some turned out to use a
rather standard video
format (https://blog.danman.eu/new-version-of-lenkeng-hdmi-over-ip-extender-lkv373a/)
for streaming. It worked fine… with one little inconvenience: handling the
network stream put a significant load on the Raspberry Pi that handled it. I
could use a different system for video processing than the RPi responsible for
USB emulation, but that would make the whole setup even more complex. Anyway,
that’s just a minor inconvenience that requires some more cooling on the RPi,
not a deal breaker.
About the time I got all of this working, I came across
PiKVM (https://pikvm.org/), which looked almost exactly like what I needed. It
uses a TC358743 chip connected directly to an RPi (via camera interface)
instead of a separate HDMI-to-IP encoder. Setting it up presented some
challenges, but the PiKVM project (or, I should say, Maxim Davaev, the guy
behind the project) had all of this figured out already.
The first issue I encountered was getting a TC358743 device initialized and
detected at all. There were several parts to this:
The default kernel from OpenSUSE does not include all the necessary drivers
(in particular, bcm2835-unicam). They’re currently available only in a
kernel from the Raspberry Pi
Foundation (https://www.raspberrypi.com/documentation/accessories/camera.html#v4l2).
I chose to compile it myself with a config based on the one from the PiKVM
project. There could be something I’m missing here, but this approach got me
a working setup, and I didn’t want to spend too much time on debugging video
drivers.
Several modifications to config.txt were required:
dtoverlay=tc358743 — let the kernel know where the device is
start_x=1 — load GPU firmware with video input processing included
gpu_mem=128 — required by start_x=1
The latter two must be in config.txt specifically, not a file
included (https://www.raspberrypi.com/documentation/computers/config_txt.html#include)
from there, which is a bit problematic on OpenSUSE, because config.txt is
forcefully overridden on each update and only the included extraconfig.txt
is meant for user modification. I worked around the issue by mounting the
bootloader partition under an alternative mountpoint to disarm the
config.txt override. This issue is in OpenSUSE’s bug
tracker (https://bugzilla.opensuse.org/show_bug.cgi?id=1192047). I have yet
to test the upstream fix for the issue.
After fixing the above, I had a /dev/video0 device. Then, it was just a
matter of configuring
it (https://forums.raspberrypi.com/viewtopic.php?f=38&t=281972). Specifically:
Loading an appropriate EDID: v4l2-ctl --set-edid=.... The EDID describes
the capabilities of this “monitor”. There is a catch if you want to use Full
HD resolution: the interface bandwidth is a bit too low for 1920x1080 with a
60Hz refresh rate, but it is enough for 50Hz (yes, unfortunately). This had
to be described in the EDID. The author of the tutorial linked above
provided some examples (https://github.com/6by9/CSI2_device_config).
Setting digital video timings: v4l2-ctl --set-dv-bt-timings query. This
can be done only when the system connected to the HDMI port starts and
chooses a resolution, and it needs to be repeated each time the resolution
changes.
I’ve integrated (https://github.com/os-autoinst/os-autoinst/pull/1741) both of
the above into the openQA driver.
For the openQA integration, using 1920x1080 resolution was not perfect. OpenQA
operates on images at 1024x768. If it receives anything else, it scales it. The
result of a 1920x1080 screen capture downscaled to 1024x768 was not nice, to
put it mildly. It not only made some text unreadable, but the difference in
aspect ratios heavily distorted the image. For example, this made it impossible
to reuse reference images made in other tests. I am considering enhancing
I started experimenting with HDMI-over-IP extenders. Some turned out to use a
rather standard video
format (https://blog.danman.eu/new-version-of-lenkeng-hdmi-over-ip-extender-lkv373a/)
for streaming. It worked fine… with one little inconvenience: handling the
network stream put a significant load on the Raspberry Pi that handled it. I
could use a different system for video processing than the RPi responsible for
USB emulation, but that would make the whole setup even more complex. Anyway,
that’s just a minor inconvenience that requires some more cooling on the RPi,
not a deal breaker.
About the time I got all of this working, I came across
PiKVM (https://pikvm.org/), which looked almost exactly like what I needed. It
uses a TC358743 chip connected directly to an RPi (via camera interface)
instead of a separate HDMI-to-IP encoder. Setting it up presented some
challenges, but the PiKVM project (or, I should say, Maxim Davaev, the guy
behind the project) had all of this figured out already.
The first issue I encountered was getting a TC358743 device initialized and
detected at all. There were several parts to this:
The default kernel from OpenSUSE does not include all the necessary drivers
(in particular, bcm2835-unicam). They’re currently available only in a
kernel from the Raspberry Pi
Foundation (https://www.raspberrypi.com/documentation/accessories/camera.html#v4l2).
I chose to compile it myself with a config based on the one from the PiKVM
project. There could be something I’m missing here, but this approach got me
a working setup, and I didn’t want to spend too much time on debugging video
drivers.
Several modifications to config.txt were required:
dtoverlay=tc358743 — let the kernel know where the device is
start_x=1 — load GPU firmware with video input processing included
gpu_mem=128 — required by start_x=1
The latter two must be in config.txt specifically, not a file
included (https://www.raspberrypi.com/documentation/computers/config_txt.html#include)
from there, which is a bit problematic on OpenSUSE, because config.txt is
forcefully overridden on each update and only the included extraconfig.txt
is meant for user modification. I worked around the issue by mounting the
bootloader partition under an alternative mountpoint to disarm the
config.txt override. This issue is in OpenSUSE’s bug
tracker (https://bugzilla.opensuse.org/show_bug.cgi?id=1192047). I have yet
to test the upstream fix for the issue.
After fixing the above, I had a /dev/video0 device. Then, it was just a
matter of configuring
it (https://forums.raspberrypi.com/viewtopic.php?f=38&t=281972). Specifically:
Loading an appropriate EDID: v4l2-ctl --set-edid=.... The EDID describes
the capabilities of this “monitor”. There is a catch if you want to use Full
HD resolution: the interface bandwidth is a bit too low for 1920x1080 with a
60Hz refresh rate, but it is enough for 50Hz (yes, unfortunately). This had
to be described in the EDID. The author of the tutorial linked above
provided some examples (https://github.com/6by9/CSI2_device_config).
Setting digital video timings: v4l2-ctl --set-dv-bt-timings query. This
can be done only when the system connected to the HDMI port starts and
chooses a resolution, and it needs to be repeated each time the resolution
changes.
I’ve integrated (https://github.com/os-autoinst/os-autoinst/pull/1741) both of
the above into the openQA driver.
For the openQA integration, using 1920x1080 resolution was not perfect. OpenQA
operates on images at 1024x768. If it receives anything else, it scales it. The
result of a 1920x1080 screen capture downscaled to 1024x768 was not nice, to
put it mildly. It not only made some text unreadable, but the difference in
aspect ratios heavily distorted the image. For example, this made it impossible
to reuse reference images made in other tests. I am considering enhancing
openQA to support other resolutions too, but for now I have set the resolution
on the tested system to 1024x768 (and used an EDID that lists that resolution).
on the tested system to 1024x768 (and used an EDID that lists that resolution).
On the test system, something needs to actually enable HDMI output. For this
purpose, I passed a kickstart file to the Qubes OS installer that includes
commands to execute before installation (the %pre section). While at it, I
could use the same kickstart file for other test-related customizations, like
restoring the default boot order at the end or enabling SSH access for
collecting logs.
HID input
Recording video output is not everything. To run tests, one also needs to send
commands to the system under test (SUT). This can be done in several ways,
including via serial console and SSH connection. In order to have the most
realistic setup, I chose to emulate USB input devices. With this, we could
interact with the system in the same way a user would. To emulate USB input
device(s), I used the Linux USB Gadget subsystem. To emulate HID devices, I had
to prepare a HID denoscriptor — a denoscription for the driver, listing what kind
of device it is and what events it can send.
I wanted the device(s) to meet the following requirements:
Have two interfaces (which in practice is two separate HID devices): keyboard
and pointer (mouse/tablet)
Be properly categorized by udev (so the input
proxy (https://github.com/QubesOS/qubes-app-linux-input-proxy) picks it up
properly)
Be properly categorized by Xorg
Support both absolute pointer position events (like “move mouse to a specific
point” instead of “move mouse a bit to the right”) and normal mouse buttons
I searched for a denoscriptor meeting the above requirements. The one for
keyboards is rather standard, but the one for mouse/tablet devices is not. So,
I took the Device Class Definition for HID 1.11 together with HID Usage Tables
1.22 (https://www.usb.org/hid) and crafted one myself. This was a bit of a
challenge, because both udev and Xorg have a set of heuristics to categorize
devices, and they differ in subtle ways.
Then, I wrote a
noscript (https://gist.github.com/marmarek/5c44ffeb2f36e106b4d34e7e8780c208)
that sets this all up and controls the device(s) according to what openQA
requests.
The last detail is about connecting an RPi to the target system. The RPi4 has a
single USB-C port used both for powering the RPi itself as well as for USB
device emulation. Generally, this would be fine, with the exception that the
target system is going to be disconnected from power from time to time. If the
RPi were powered this way, it would lose power too, and there would be nothing
capable of turning it back on. This is yet another case where the PiKVM project
provided an inspiration (https://github.com/pikvm/pikvm#setting-up-the-v2): a
Y-split cable that connects the VBUS pin to only one end and the data pins to
the other.
Serial console
Several openQA functions require some kind of console access. This includes
retrieving command outputs (and exit codes), waiting for various events, etc.
Unfortunately, a real serial console is very rare in modern laptops. I could
restructure the tests not to use those functions, but that would be rather
disappointing in terms of test result quality. As a solution, I added a small
qrexec service in dom0 that reads a pipe that pretends to be a serial console,
then I used qvm-connect-tcp in sys-net to redirect the TCP port to that
service. This isn’t as reliable as a real serial console (especially for things
like restarting sys-net), but it does work in the majority of cases. In the
future, I will restructure the tests not to rely on this functionality in order
to account for the few rare cases where it doesn’t work.
Bonus: remote-controlled test laptops for developers
Remote power and boot control is useful not only for automatic tests, but also
for ordinary developers. There are several cases where it is useful:
Additional machines to develop and test features on different versions of
Qubes
Access to specific hardware
I’ve prepared the whole setup to be usable not only with openQA, but also to
purpose, I passed a kickstart file to the Qubes OS installer that includes
commands to execute before installation (the %pre section). While at it, I
could use the same kickstart file for other test-related customizations, like
restoring the default boot order at the end or enabling SSH access for
collecting logs.
HID input
Recording video output is not everything. To run tests, one also needs to send
commands to the system under test (SUT). This can be done in several ways,
including via serial console and SSH connection. In order to have the most
realistic setup, I chose to emulate USB input devices. With this, we could
interact with the system in the same way a user would. To emulate USB input
device(s), I used the Linux USB Gadget subsystem. To emulate HID devices, I had
to prepare a HID denoscriptor — a denoscription for the driver, listing what kind
of device it is and what events it can send.
I wanted the device(s) to meet the following requirements:
Have two interfaces (which in practice is two separate HID devices): keyboard
and pointer (mouse/tablet)
Be properly categorized by udev (so the input
proxy (https://github.com/QubesOS/qubes-app-linux-input-proxy) picks it up
properly)
Be properly categorized by Xorg
Support both absolute pointer position events (like “move mouse to a specific
point” instead of “move mouse a bit to the right”) and normal mouse buttons
I searched for a denoscriptor meeting the above requirements. The one for
keyboards is rather standard, but the one for mouse/tablet devices is not. So,
I took the Device Class Definition for HID 1.11 together with HID Usage Tables
1.22 (https://www.usb.org/hid) and crafted one myself. This was a bit of a
challenge, because both udev and Xorg have a set of heuristics to categorize
devices, and they differ in subtle ways.
Then, I wrote a
noscript (https://gist.github.com/marmarek/5c44ffeb2f36e106b4d34e7e8780c208)
that sets this all up and controls the device(s) according to what openQA
requests.
The last detail is about connecting an RPi to the target system. The RPi4 has a
single USB-C port used both for powering the RPi itself as well as for USB
device emulation. Generally, this would be fine, with the exception that the
target system is going to be disconnected from power from time to time. If the
RPi were powered this way, it would lose power too, and there would be nothing
capable of turning it back on. This is yet another case where the PiKVM project
provided an inspiration (https://github.com/pikvm/pikvm#setting-up-the-v2): a
Y-split cable that connects the VBUS pin to only one end and the data pins to
the other.
Serial console
Several openQA functions require some kind of console access. This includes
retrieving command outputs (and exit codes), waiting for various events, etc.
Unfortunately, a real serial console is very rare in modern laptops. I could
restructure the tests not to use those functions, but that would be rather
disappointing in terms of test result quality. As a solution, I added a small
qrexec service in dom0 that reads a pipe that pretends to be a serial console,
then I used qvm-connect-tcp in sys-net to redirect the TCP port to that
service. This isn’t as reliable as a real serial console (especially for things
like restarting sys-net), but it does work in the majority of cases. In the
future, I will restructure the tests not to rely on this functionality in order
to account for the few rare cases where it doesn’t work.
Bonus: remote-controlled test laptops for developers
Remote power and boot control is useful not only for automatic tests, but also
for ordinary developers. There are several cases where it is useful:
Additional machines to develop and test features on different versions of
Qubes
Access to specific hardware
I’ve prepared the whole setup to be usable not only with openQA, but also to
👍2🤮1
allow for the delegation of specific test machines to individual trusted
developers. More importantly, this allows not only for manually testing
software on those machines, but also for automating several tasks, such as the
Git bisection mentioned earlier.
Final thoughts
Testing a whole operating system is a challenging task, because there are a lot
of moving parts. OpenQA is a great tool for that, but its main target is
running tests in a virtualized environment. This works fine for several
components (like Qubes Manager and GUI virtualization) but not for
hardware-related features (sys-net, sys-usb, system suspend, and several
others). Before this work, we ran tests on actual laptops manually, but that
was time-consuming and thus not all updates or configurations were tested.
Automation allows our testing to be much more comprehensive, including ensuring
ongoing compatibility with Qubes OS certified hardware.
developers. More importantly, this allows not only for manually testing
software on those machines, but also for automating several tasks, such as the
Git bisection mentioned earlier.
Final thoughts
Testing a whole operating system is a challenging task, because there are a lot
of moving parts. OpenQA is a great tool for that, but its main target is
running tests in a virtualized environment. This works fine for several
components (like Qubes Manager and GUI virtualization) but not for
hardware-related features (sys-net, sys-usb, system suspend, and several
others). Before this work, we ran tests on actual laptops manually, but that
was time-consuming and thus not all updates or configurations were tested.
Automation allows our testing to be much more comprehensive, including ensuring
ongoing compatibility with Qubes OS certified hardware.
👍3👏3🤮1
Fedora 34 approaching EOL; Fedora 35 templates available
https://www.qubes-os.org/news/2022/05/26/fedora-34-approaching-eol-fedora-35-templates-available/
Fedora 34 is scheduled to reach EOL (end-of-life (https://fedoraproject.org/wiki/End_of_life)) on 2022-06-07, and
new Fedora 35 templates are now available for both Qubes 4.0 and 4.1.
We strongly recommend that all Qubes users upgrade (https://www.qubes-os.org/doc/templates/fedora/#upgrading) their Fedora 34
templates and standalones to Fedora 35 before Fedora 34 reaches EOL.
We provide fresh Fedora 35 template packages through the official Qubes
repositories, which you can install in dom0 by following the standard
installation instructions (https://www.qubes-os.org/doc/templates/fedora/#installing). Alternatively, we also provide step-by-step
instructions for performing an in-place upgrade (https://www.qubes-os.org/doc/templates/fedora/) of an existing Fedora
template. After upgrading your templates, please remember to switch all
qubes that were using the old template to use the new one (https://www.qubes-os.org/doc/templates/#switching).
For a complete list of template releases that are supported for your
specific Qubes release, see our supported template releases (https://www.qubes-os.org/doc/supported-releases/#templates).
Please note that no user action is required regarding the OS version in
dom0. For details, please see our note on dom0 and EOL (https://www.qubes-os.org/doc/supported-releases/#note-on-dom0-and-eol).
https://www.qubes-os.org/news/2022/05/26/fedora-34-approaching-eol-fedora-35-templates-available/
Fedora 34 is scheduled to reach EOL (end-of-life (https://fedoraproject.org/wiki/End_of_life)) on 2022-06-07, and
new Fedora 35 templates are now available for both Qubes 4.0 and 4.1.
We strongly recommend that all Qubes users upgrade (https://www.qubes-os.org/doc/templates/fedora/#upgrading) their Fedora 34
templates and standalones to Fedora 35 before Fedora 34 reaches EOL.
We provide fresh Fedora 35 template packages through the official Qubes
repositories, which you can install in dom0 by following the standard
installation instructions (https://www.qubes-os.org/doc/templates/fedora/#installing). Alternatively, we also provide step-by-step
instructions for performing an in-place upgrade (https://www.qubes-os.org/doc/templates/fedora/) of an existing Fedora
template. After upgrading your templates, please remember to switch all
qubes that were using the old template to use the new one (https://www.qubes-os.org/doc/templates/#switching).
For a complete list of template releases that are supported for your
specific Qubes release, see our supported template releases (https://www.qubes-os.org/doc/supported-releases/#templates).
Please note that no user action is required regarding the OS version in
dom0. For details, please see our note on dom0 and EOL (https://www.qubes-os.org/doc/supported-releases/#note-on-dom0-and-eol).
🎉3