WIP: Make CPU profiles MSR aware#70
Closed
olivereanderson wants to merge 23 commits intocyberus-technology:gardenlinux-v50from
Closed
WIP: Make CPU profiles MSR aware#70olivereanderson wants to merge 23 commits intocyberus-technology:gardenlinux-v50from
olivereanderson wants to merge 23 commits intocyberus-technology:gardenlinux-v50from
Conversation
7a0c713 to
abe10dd
Compare
324135c to
b8fa2a0
Compare
This was referenced Feb 9, 2026
b85447c to
1463253
Compare
4bf173a to
80bdbe5
Compare
74532c2 to
dd3d9b6
Compare
Member
|
please change the PR target to the new gardenlinux branch |
dd3d9b6 to
c723354
Compare
In order to apply CPU profiles we need to modify the MSR-based features according to the profile. The first step is to obtain these MSR-based features from the hypervisor as introduced in this commit. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We want to deny MSRs that are not available, or should not be available for a given CPU profile, but in order to do that we must also have some means to see which MSRs are supported by the host. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We realized that we only need a method for obtaining supported MSR indices and the buffer can be built from that in the `CpuManager`'s constructor. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
In order to ensure that MSRs that are not compatible with a given CPU profile do no get accessed by the guests we need to introduce functionality to deny such MSRs via filters. The implementation introduced here is mostly a temporary workaround until rust-vmm/kvm#359 is integrated in CHV. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We decided to disable UINTR for CPU profiles Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We disable CET through CPUID for CPU profiles, because it is currently not supported in QEMU. We might want to revisit this decission. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We need to change the policies for MCA and MCE for non-host CPU profiles. By setting the profile policy to Static(0) for the MCA bit we indicate to guests that the MCG_CAP MSR is not avaiable. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We change the CPUID policy for WAITPKG because we encountered problems with it when testing CPU profiles with MSRs. This is also off by default for CPU models in QEMU, but we may still potentially want to revisit this decision in the future. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
The KVM_CLOCKSOURCE CPUID bit informs guets that MSRs (KVM_WALL_CLOCK) and (KVM_SYSTEM_TIME) are available. However these MSRs fall outside the reserved KVM MSR range and are deprecated. Since replacements exist (MSR_KVM_WALL_CLOCK_NEW) and also (MSR_KVM_SYSTEM_TIME_NEW) we decide to zero out the CPUID feature bit indicating support for the deprecated MSRs. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
In preparation for making CPU profiles MSR aware we prepare for having two pieces of data associated with a CPU profile: CPUID and MSR adjustments. We thus rename the pre-existing CpuProfileData struct to CpuIdProfileData and adapt the CPU profile generation tool accordingly. We also make the CPU profile generation tool write directly to file and automatically introduce the required license file as well. This makes the profile generation process more convenient. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
Introduce convenience methods for looking up CPUID value definitions. We will later use these methods to assert certain policies at compile time in order to stay consistent with MSR policies we introduce. Due to the current limitations of const generics we unfortunately need to duplicate a little bit code. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
In order to generate CPU profiles we also need definitions and policies for MSR-based features, as some CPU features are exposed through MSRs rather than CPUID. This commit introduces the MSR analogues of the data structures we previously introduced for CPUID definitions. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We introduce MSR-based feature definitions for Intel CPUs that will be utilized by the upcoming CPU profile generation tool. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
While KVM already has compatibility checks for most MSR-based features which run when these MSRs are set by userspace, we do not get very much useful information about exactly what the problem is when any of these checks fail. Hence to be on the safe side and also to ensure good UX for users running into errors when trying to apply a CPU profile we introduce our own compatibility checks for Intel CPUs that log at the error and debug levels. The error logs aim to provide the minimal amount of information required to investigate the problem further, while the debug logs provide (much) more convenience when debugging. We will incorporate these checks in the context of CPU profiles in a later commit. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
This list will be used to help us detect unknown MSRs when generating CPU profiles. It serves no other purpose beyond that. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We include a list of non-architectural MSRS. This list will only be used to help the CPU profile generation tool rule out MSRs that it does not know how to handle. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We include a list of MSRS defined by KVM that may be approved by CPU profiles and another list of those that may not be approved by CPU profiles. These lists will later be used by the CPU profile generation tool. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
The list of HyperV MSRs introduced here will be utilized during CPU profile generation and also at runtime to filter them out whenever `kvm_hyperv` is set to `false`. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We introduce functionality related to computing necessary MSR updates in accordance with the given CPU profile. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We introduce functionality to filter out MSRs which we want to deny guests from using. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We record the necessary MSR-based feature modifications that need to be set in the `CpuManager` and make sure to set these MSR values upon vCPU configuration. We also use the Vm to filter access to MSRs that are incompatible with the chosen CPU profile. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We adapt the CPU profile generation tool to also take the MSR-based features into account. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
We regenerate the CPU profiles and include the MSR-related data. Signed-off-by: Oliver Anderson <oliver.anderson@cyberus-technology.de> On-behalf-of: SAP oliver.anderson@sap.com
Author
|
closed in favour of #103 |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds functionality to modify and restrict MSRs based on CPU profiles.
Hints for reviewers: Please read the following description below in its entirety and then review this commit for commit.
Description
This PR continues the work merged in #62. If you do not recall the details from that PR I suggest reading the PR description again before continuing.
MSR-based features
Most CPU features can indeed be probed through CPUID, but there are a few that instead require MSR inspection.
We follow KVM and name these MSR-based features (See KVM_GET_MSR_FEATURE_INDEX_LIST).
Up to this point in time a similar Time-of-check-to-time-of-use bug as described in the context of CPUID in #62 can occur in the context of MSRs. Indeed suppose for example that a guest checks
IA32_VMX_VMCS_ENUMto find the highest index value used for any VMCS encoding and then a live migration to a machine with a lower value occurs.Even though KVM will (presumably) catch this potential error when attempting to set MSRs on the migration destination, it still means that live migration can never take place between these machines.
Hence in order to force a higher level of compatibility we take advantage of the fact that KVM can lie to guests about MSRs values in an analogous manner to the CPUID case.
We thus extend the CPU profile generation tool to also record necessary adjustments to the MSR-based feature values, using profile policies similar to those we previously introduced in the CPUID context.
MSR filters
At this point in time there are less than 40 MSR-based features that KVM will return when calling
KVM_GET_MSR_FEATURE_INDEX_LISTon any given Intel CPU, but this is only a small fraction of the number of MSRs that are actually supported by the hardware and/or hypervisor.While many MSRs are described as only being accessible if certain CPUID bits/values are present, there are some that are available on all CPUs of introduced after a certain generation (without CPUID requirements), and even some MSRs that are only available on specific CPUs. The latter are referred to as non-architectural msrs. MSRs whose values (although possibly different) have the same definitions across processor generations are called architectural MSRS (prefixed with
IA32).To combat the problem of MSRs that may only be available on the source VM, but not on the destination VM,
we take advantage of KVM's
KVM_X86_SET_MSR_FILTERwhich enables us to deny guests from accessing entire ranges of MSRs.More precisely we record a subset of the architectural MSRs supported by the hardware and hypervisor into the CPU profile and set up a filter to deny any MSRs not listed there when the CPU profile is applied.
Note that it is possible to perform even more fine grained MSR filtering (only denying individual bits) with
KVM_CAP_X86_USER_SPACE_MSR, but we don't think that is necessary in our case.CPUID adjustments
Some of the commits here are related to CPUID profile policy adjustments because they turned out to be problematic when testing the CPU profiles after adding MSR adjustments.
Making the CPU profile generation tool future proof
The whole point of the CPU profile generation tool is to automate the process of creating CPU profiles for new CPUs. Since new CPU generations (or KVM versions) may introduce new architectural MSRs (possibly even MSR-based features) that we are not yet aware of, we make the CPU profile tool emit warnings whenever it encounters MSRs that it is not aware of.
In this way we get notified when it might be time to update the MSR definitions used by the CPU profile tool. In order to achieve this we have introduced somewhat long lists of MSRs that we know about at this point in time. They have no other purpose other than helping us keep the CPU profile generation tool up to date!
Note that bits within already existing MSRs might of course go from being reserved to defined in the future and we do not know of a good way to automatically detect that (especially since the reserved value is not necessarily 0 in every case).