Skip to content

WIP: Make CPU profiles MSR aware#70

Closed
olivereanderson wants to merge 23 commits intocyberus-technology:gardenlinux-v50from
olivereanderson:cpu-profiles-msr
Closed

WIP: Make CPU profiles MSR aware#70
olivereanderson wants to merge 23 commits intocyberus-technology:gardenlinux-v50from
olivereanderson:cpu-profiles-msr

Conversation

@olivereanderson
Copy link

@olivereanderson olivereanderson commented Jan 26, 2026

This PR adds functionality to modify and restrict MSRs based on CPU profiles.

Hints for reviewers: Please read the following description below in its entirety and then review this commit for commit.

Description

This PR continues the work merged in #62. If you do not recall the details from that PR I suggest reading the PR description again before continuing.

MSR-based features

Most CPU features can indeed be probed through CPUID, but there are a few that instead require MSR inspection.
We follow KVM and name these MSR-based features (See KVM_GET_MSR_FEATURE_INDEX_LIST).

Up to this point in time a similar Time-of-check-to-time-of-use bug as described in the context of CPUID in #62 can occur in the context of MSRs. Indeed suppose for example that a guest checks IA32_VMX_VMCS_ENUM to find the highest index value used for any VMCS encoding and then a live migration to a machine with a lower value occurs.
Even though KVM will (presumably) catch this potential error when attempting to set MSRs on the migration destination, it still means that live migration can never take place between these machines.

Hence in order to force a higher level of compatibility we take advantage of the fact that KVM can lie to guests about MSRs values in an analogous manner to the CPUID case.

We thus extend the CPU profile generation tool to also record necessary adjustments to the MSR-based feature values, using profile policies similar to those we previously introduced in the CPUID context.

MSR filters

At this point in time there are less than 40 MSR-based features that KVM will return when calling KVM_GET_MSR_FEATURE_INDEX_LIST on any given Intel CPU, but this is only a small fraction of the number of MSRs that are actually supported by the hardware and/or hypervisor.

While many MSRs are described as only being accessible if certain CPUID bits/values are present, there are some that are available on all CPUs of introduced after a certain generation (without CPUID requirements), and even some MSRs that are only available on specific CPUs. The latter are referred to as non-architectural msrs. MSRs whose values (although possibly different) have the same definitions across processor generations are called architectural MSRS (prefixed with IA32).

To combat the problem of MSRs that may only be available on the source VM, but not on the destination VM,
we take advantage of KVM's KVM_X86_SET_MSR_FILTER which enables us to deny guests from accessing entire ranges of MSRs.

More precisely we record a subset of the architectural MSRs supported by the hardware and hypervisor into the CPU profile and set up a filter to deny any MSRs not listed there when the CPU profile is applied.

Note that it is possible to perform even more fine grained MSR filtering (only denying individual bits) with KVM_CAP_X86_USER_SPACE_MSR, but we don't think that is necessary in our case.

CPUID adjustments

Some of the commits here are related to CPUID profile policy adjustments because they turned out to be problematic when testing the CPU profiles after adding MSR adjustments.

Making the CPU profile generation tool future proof

The whole point of the CPU profile generation tool is to automate the process of creating CPU profiles for new CPUs. Since new CPU generations (or KVM versions) may introduce new architectural MSRs (possibly even MSR-based features) that we are not yet aware of, we make the CPU profile tool emit warnings whenever it encounters MSRs that it is not aware of.

In this way we get notified when it might be time to update the MSR definitions used by the CPU profile tool. In order to achieve this we have introduced somewhat long lists of MSRs that we know about at this point in time. They have no other purpose other than helping us keep the CPU profile generation tool up to date!

Note that bits within already existing MSRs might of course go from being reserved to defined in the future and we do not know of a good way to automatically detect that (especially since the reserved value is not necessarily 0 in every case).

@olivereanderson olivereanderson force-pushed the cpu-profiles-msr branch 7 times, most recently from 7a0c713 to abe10dd Compare February 3, 2026 15:17
@olivereanderson olivereanderson force-pushed the cpu-profiles-msr branch 5 times, most recently from 324135c to b8fa2a0 Compare February 9, 2026 19:21
@olivereanderson olivereanderson force-pushed the cpu-profiles-msr branch 6 times, most recently from b85447c to 1463253 Compare February 13, 2026 23:21
@olivereanderson olivereanderson force-pushed the cpu-profiles-msr branch 10 times, most recently from 4bf173a to 80bdbe5 Compare February 24, 2026 13:44
@phip1611 phip1611 self-requested a review March 6, 2026 15:13
@olivereanderson olivereanderson force-pushed the cpu-profiles-msr branch 2 times, most recently from 74532c2 to dd3d9b6 Compare March 6, 2026 15:32
@phip1611
Copy link
Member

phip1611 commented Mar 6, 2026

please change the PR target to the new gardenlinux branch

In order to apply CPU profiles we need to modify the MSR-based features
according to the profile. The first step is to obtain these MSR-based
features from the hypervisor as introduced in this commit.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We want to deny MSRs that are not available, or should not be available
for a given CPU profile, but in order to do that we must also have
some means to see which MSRs are supported by the host.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We realized that we only need a method for obtaining supported MSR
indices and the buffer can be built from that in the `CpuManager`'s
constructor.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
In order to ensure that MSRs that are not compatible with a given
CPU profile do no get accessed by the guests we need to introduce
functionality to deny such MSRs via filters.

The implementation introduced here is mostly a temporary workaround
until rust-vmm/kvm#359 is integrated in
CHV.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We decided to disable UINTR for CPU profiles

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We disable CET through CPUID for CPU profiles, because it is currently
not supported in QEMU.

We might want to revisit this decission.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We need to change the policies for MCA and MCE for non-host CPU
profiles. By setting the profile policy to Static(0) for the MCA bit
we indicate to guests that the MCG_CAP MSR is not avaiable.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We change the CPUID policy for WAITPKG because we encountered problems
with it when testing CPU profiles with MSRs.

This is also off by default for CPU models in QEMU, but we may still
potentially want to revisit this decision in the future.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
The KVM_CLOCKSOURCE CPUID bit informs guets that MSRs
(KVM_WALL_CLOCK) and (KVM_SYSTEM_TIME) are available. However
these MSRs fall outside the reserved KVM MSR range and are
deprecated. Since replacements exist (MSR_KVM_WALL_CLOCK_NEW) and also
(MSR_KVM_SYSTEM_TIME_NEW) we decide to zero out the CPUID feature bit
indicating support for the deprecated MSRs.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
In preparation for making CPU profiles MSR aware we prepare for having
two pieces of data associated with a CPU profile: CPUID and MSR
adjustments. We thus rename the pre-existing CpuProfileData struct
to CpuIdProfileData and adapt the CPU profile generation tool
accordingly.

We also make the CPU profile generation tool write directly to file
and automatically introduce the required license file as well. This
makes the profile generation process more convenient.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
Introduce convenience methods for looking up CPUID value definitions.

We will later use these methods to assert certain policies at compile
time in order to stay consistent with MSR policies we introduce.

Due to the current limitations of const generics we unfortunately
need to duplicate a little bit code.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
In order to generate CPU profiles we also need definitions and policies
for MSR-based features, as some CPU features are exposed through MSRs
rather than CPUID.

This commit introduces the MSR analogues of the data structures we
previously introduced for CPUID definitions.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We introduce MSR-based feature definitions for Intel CPUs that will be
utilized by the upcoming CPU profile generation tool.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
While KVM already has compatibility checks for most MSR-based features
which run when these MSRs are set by userspace, we do not get very
much useful information about exactly what the problem is when any
of these checks fail.

Hence to be on the safe side and also to ensure good UX for users
running into errors when trying to apply a CPU profile we introduce our
own compatibility checks for Intel CPUs that log at the error and debug
levels. The error logs aim to provide the minimal amount of information
required to investigate the problem further, while the debug logs
provide (much) more convenience when debugging.

We will incorporate these checks in the context of CPU profiles in
a later commit.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
This list will be used to help us detect unknown MSRs when generating
CPU profiles. It serves no other purpose beyond that.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We include a list of non-architectural MSRS. This list will only be
used to help the CPU profile generation tool rule out MSRs that it
does not know how to handle.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We include a list of MSRS defined by KVM that may be approved by
CPU profiles and another list of those that may not be approved by
CPU profiles. These lists will later be used by the CPU profile
generation tool.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
The list of HyperV MSRs introduced here will be utilized during CPU
profile generation and also at runtime to filter them out whenever
`kvm_hyperv` is set to `false`.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We introduce functionality related to computing necessary MSR updates
in accordance with the given CPU profile.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We introduce functionality to filter out MSRs which we want to deny
guests from using.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We record the necessary MSR-based feature modifications that need to be
set in the `CpuManager` and make sure to set these MSR values upon
vCPU configuration. We also use the Vm to filter access to MSRs that
are incompatible with the chosen CPU profile.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We adapt the CPU profile generation tool to also take the MSR-based
features into account.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
We regenerate the CPU profiles and include the MSR-related data.

Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de>
On-behalf-of: SAP oliver.anderson@sap.com
@olivereanderson
Copy link
Author

closed in favour of #103

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants