OCPCLOUD-3346: tls: use centralized TLS by RadekManak · Pull Request #1456 · openshift/machine-api-operator

RadekManak · 2026-01-26T13:17:05Z

Serve machine-api-operator metrics directly over HTTPS (:8443) using controller-runtime’s metrics server with delegated authn/authz (WithAuthenticationAndAuthorization), and remove the MAO kube-rbac-proxy sidecar.
Add TLS profile awareness for MAO metrics:
- read APIServer/cluster TLS profile on startup,
- configure min TLS/ciphers from that profile,
- watch for TLS profile changes and trigger shutdown so the pod restarts with updated TLS settings.
Propagate the same TLS profile to controller kube-rbac-proxy sidecars (machine, machineset, mhc) by generating --tls-min-version and profile-derived --tls-cipher-suites args.
Update manifests accordingly:
- deployment ports/volume mounts/env (METRICS_PORT=8443) for direct secure serving,
- RBAC to watch config.openshift.io/apiservers.
Include supporting dependency/vendor updates and minor follow-ups:
- dependency bumps (controller-runtime, openshift/api, openshift/client-go, etc.),
- go-build.sh root-dir handling fix,
- lint/import cleanup (pkg/webhooks/machine_webhook.go, context import updates).

Notes

Provider/controller metrics remain behind kube-rbac-proxy and continue using the existing namespace/metrics authorization model.
MAO direct /metrics auth uses delegated token/SAR checks; scraper access relies on existing cluster-monitoring prometheus-k8s cluster RBAC.

openshift-ci-robot · 2026-01-26T13:17:10Z

@RadekManak: This pull request references OCPCLOUD-3346 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Summary

Replace the MAO metrics kube‑rbac‑proxy sidecar with direct HTTPS in the operator binary, using the serving cert mounted from the MAO secret.

Watch the APIServer TLS profile and trigger a controlled shutdown so MAO restarts and picks up TLS changes.

Propagate the APIServer TLS profile into machine-api-controllers kube‑rbac‑proxy args (cipher suites + min TLS), with unit coverage.

Details

Direct MAO metrics TLS

MAO now listens on :8443 and serves /metrics via ListenAndServeTLS using /etc/tls/private/tls.crt|tls.key.

The deployment drops the kube‑rbac‑proxy sidecar, mounts the serving cert into /etc/tls/private, and exposes port 8443.

RBAC is updated to allow reading apiservers for TLS profile fetch.

TLS profile reload

MAO fetches the APIServer TLS profile at startup and builds a tls.Config.

A config informer watches APIServer updates and triggers shutdown on profile changes.

Centralized proxy TLS for controllers

OperatorConfig now carries the TLS profile.

machine-api-controllers kube‑rbac‑proxy args are generated from the profile (--tls-cipher-suites, --tls-min-version),

Tests updated to include APIServer presence and TLS profile expectations; a focused test validates proxy TLS args.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

RadekManak · 2026-01-26T13:17:18Z

/assign @damdo

damdo

Thanks! Looks good, a couple of minor points.

damdo · 2026-01-26T13:30:03Z

pkg/operator/operator.go

+	var tlsProfile *osconfigv1.TLSProfileSpec
+	apiServer, err := optr.osClient.ConfigV1().APIServers().Get(context.Background(), "cluster", metav1.GetOptions{})
+	if err != nil {
+		klog.Warningf("Failed to fetch APIServer, using default TLS profile: %v", err)


Shouldn't we return here?
What would happen otherwise?

pkg/operator/operator.go

pkg/operator/sync.go

openshift-ci · 2026-01-27T12:34:44Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by:
Once this PR has been reviewed and has the lgtm label, please ask for approval from damdo. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

Remove the kube-rbac-proxy sidecar, mount the serving cert, and restart the operator on APIServer TLS profile changes.

Capture the APIServer TLS profile in operator config and use it to configure kube-rbac-proxy TLS args, with unit coverage.

Add unit tests to verify TLS configuration handling in newKubeProxyContainer, including tests for TLS 1.2 with cipher suites and TLS 1.3 without cipher suites.

openshift-ci-robot · 2026-02-18T15:05:46Z

@RadekManak: This pull request references OCPCLOUD-3346 which is a valid jira issue.

Warning: The referenced jira issue has an invalid target version for the target branch this PR targets: expected the story to target the "4.22.0" version, but no target version was set.

Details

In response to this:

Serve machine-api-operator metrics directly over HTTPS (:8443) using controller-runtime’s metrics server with delegated authn/authz (WithAuthenticationAndAuthorization), and remove the MAO kube-rbac-proxy sidecar.

Add TLS profile awareness for MAO metrics:

read APIServer/cluster TLS profile on startup,

configure min TLS/ciphers from that profile,

watch for TLS profile changes and trigger shutdown so the pod restarts with updated TLS settings.

Propagate the same TLS profile to controller kube-rbac-proxy sidecars (machine, machineset, mhc) by generating --tls-min-version and profile-derived --tls-cipher-suites args.

Update manifests accordingly:

deployment ports/volume mounts/env (METRICS_PORT=8443) for direct secure serving,

RBAC to watch config.openshift.io/apiservers.

Include supporting dependency/vendor updates and minor follow-ups:

dependency bumps (controller-runtime, openshift/api, openshift/client-go, etc.),

go-build.sh root-dir handling fix,

lint/import cleanup (pkg/webhooks/machine_webhook.go, context import updates).

Notes

Provider/controller metrics remain behind kube-rbac-proxy and continue using the existing namespace/metrics authorization model.

MAO direct /metrics auth uses delegated token/SAR checks; scraper access relies on existing cluster-monitoring prometheus-k8s cluster RBAC.

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the openshift-eng/jira-lifecycle-plugin repository.

damdo

Thanks for this Radek!
I left a bunch of comments but it mostly looks good! TY

pkg/operator/sync.go

damdo · 2026-02-23T10:38:15Z

cmd/machine-api-operator/start.go

+		return
 	}
-	klog.Fatal(server.ListenAndServe())
+


In controller-runtime-common we overwrite initialProfile once the change is detected, see: https://github.com/damdo/machine-api-operator/blob/6b35a05c94ac5760bedfcb4e2408d8efee3d1100/vendor/github.com/openshift/controller-runtime-common/pkg/tls/controller.go#L138-L140

We are restarting anyway so it does not really matter, but I agree that is more correct if we got multiple updates before shutdown. This way, the second event won't log the shutdown again.

cmd/machine-api-operator/start.go

damdo · 2026-02-23T10:40:31Z

cmd/machine-api-operator/start.go

+	return metricsserver.NewServer(metricsserver.Options{
+		BindAddress:    metricsAddr,
+		SecureServing:  true,
+		FilterProvider: filters.WithAuthenticationAndAuthorization,


Do we have the right RBAC defined for the MAO Deployment's service account to do this?

WithAuthenticationAndAuthorization provides a metrics.Filter for authentication and authorization. Metrics will be authenticated (via TokenReviews) and authorized (via SubjectAccessReviews) with the kube-apiserver. For the authentication and authorization the controller needs a ClusterRole with the following rules: * apiGroups: authentication.k8s.io, resources: tokenreviews, verbs: create * apiGroups: authorization.k8s.io, resources: subjectaccessreviews, verbs: create
To scrape metrics e.g. via Prometheus the client needs a ClusterRole with the following rule: * nonResourceURLs: "/metrics", verbs: get

https://github.com/openshift/machine-api-operator/blob/main/install/0000_30_machine-api-operator_09_rbac.yaml#L399-L406

Yes, we already have this.

Move TLS configuration computation from per-container to once per batch in newKubeProxyContainers. This avoids redundant processing when creating multiple kube-rbac-proxy containers with the same TLS profile.

openshift-ci · 2026-02-23T19:43:25Z

@RadekManak: The following test failed, say /retest to rerun all failed tests or /retest-required to rerun all mandatory failed tests:

Test name	Commit	Details	Required	Rerun command
ci/prow/e2e-metal-ipi-ovn-ipv6	`5d8f69d`	link	true	`/test e2e-metal-ipi-ovn-ipv6`

Full PR test history. Your PR dashboard.

Details

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository. I understand the commands that are listed here.

openshift-ci-robot added the jira/valid-reference Indicates that this PR references a valid Jira ticket of any type. label Jan 26, 2026

openshift-ci bot assigned damdo Jan 26, 2026

openshift-ci bot requested review from damdo and theobarberbany January 26, 2026 13:18

damdo reviewed Jan 26, 2026

View reviewed changes

RadekManak added 4 commits February 18, 2026 13:34

Serve MAO metrics with direct TLS and profile reload.

a92658e

Remove the kube-rbac-proxy sidecar, mount the serving cert, and restart the operator on APIServer TLS profile changes.

Propagate TLS profile to controller proxies.

b2d8c8b

Capture the APIServer TLS profile in operator config and use it to configure kube-rbac-proxy TLS args, with unit coverage.

Fix build script

90cae05

Add tests for newKubeProxyContainer

e1d04fc

Add unit tests to verify TLS configuration handling in newKubeProxyContainer, including tests for TLS 1.2 with cipher suites and TLS 1.3 without cipher suites.

RadekManak force-pushed the feature-centralized-tsl-endpoints branch from 4e665c0 to 24eed11 Compare February 18, 2026 14:55

RadekManak added 3 commits February 18, 2026 16:25

Vendor

9b6eee5

Use controller-runtime metrics server auth filter.

72fe4c6

Fix lint

6b35a05

RadekManak force-pushed the feature-centralized-tsl-endpoints branch from 24eed11 to 6b35a05 Compare February 18, 2026 15:25

damdo reviewed Feb 23, 2026

View reviewed changes

RadekManak added 2 commits February 23, 2026 15:34

Persist TLS profile for change detection

75dc48e

Compute TLS arguments once for kube-rbac-proxy containers

5d8f69d

Move TLS configuration computation from per-container to once per batch in newKubeProxyContainers. This avoids redundant processing when creating multiple kube-rbac-proxy containers with the same TLS profile.

Comments

Conversation

RadekManak commented Jan 26, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes

Uh oh!

openshift-ci-robot commented Jan 26, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Details

Uh oh!

RadekManak commented Jan 26, 2026

Uh oh!

damdo left a comment

Choose a reason for hiding this comment

Uh oh!

damdo Jan 26, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Uh oh!

openshift-ci bot commented Jan 27, 2026

Uh oh!

openshift-ci-robot commented Feb 18, 2026 • edited by openshift-ci bot Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Notes

Uh oh!

damdo left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

damdo Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

RadekManak Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

Uh oh!

damdo Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

RadekManak Feb 23, 2026

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Feb 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

RadekManak commented Jan 26, 2026 •

edited

Loading

openshift-ci-robot commented Jan 26, 2026 •

edited by openshift-ci bot

Loading

openshift-ci-robot commented Feb 18, 2026 •

edited by openshift-ci bot

Loading