:bug: Operator operator-lifecycle-manager-packageserver is in Available=False state running in single-replica topology by jianzhangbjz · Pull Request #3720 · operator-framework/operator-lifecycle-manager

jianzhangbjz · 2025-12-11T07:39:09Z

During cluster upgrades, the operator-lifecycle-manager-packageserver ClusterOperator incorrectly reports Available=False for ~16 seconds, violating the OpenShift contract: A component must not report Available=False during the course of a normal upgrade.

PodAntiAffinity + Single-Node Cluster + Single-Replica Deployment
During rolling updates in single-node control plane environments:

Old pod is running on the only node
New pod attempts to schedule
PodAntiAffinity prevents new pod from co-locating with old pod
New pod becomes Unschedulable (waiting for old pod termination)
APIService is temporarily unavailable
BUG: Unschedulable was incorrectly treated as a real failure
CSV enters Failed phase → ClusterOperator reports Available=False

This is especially problematic in OpenShift SNO (Single Node OpenShift) environments.

Description of the change:
Enhanced pod disruption detection in pkg/controller/operators/olm/apiservices.go:

Improved rollout detection: Added checks for Generation != ObservedGeneration and AvailableReplicas < desired to catch early-phase rollouts
PodAntiAffinity awareness: Treat Unschedulable as expected disruption during single-replica rollouts instead of real failure
Single-replica tolerance: For single-replica deployments during rollout with no real failures, return RetryableError instead of marking CSV as Failed

Motivation for the change:
To address https://issues.redhat.com/browse/OCPBUGS-67210

Architectural changes:

Testing remarks:

Reviewer Checklist

Assisted-by: Claude Code

jianzhangbjz · 2025-12-11T07:49:28Z

/test sanity

openshift-ci · 2025-12-11T07:49:31Z

@jianzhangbjz: No presubmit jobs available for operator-framework/operator-lifecycle-manager@master

Details

In response to this:

/test sanity

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jianzhangbjz · 2025-12-11T07:49:52Z

/test verify

openshift-ci · 2025-12-11T07:49:56Z

@jianzhangbjz: No presubmit jobs available for operator-framework/operator-lifecycle-manager@master

Details

In response to this:

/test verify

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jianzhangbjz · 2025-12-11T07:50:17Z

/test verify (pull_request)

openshift-ci · 2025-12-11T07:50:21Z

@jianzhangbjz: No presubmit jobs available for operator-framework/operator-lifecycle-manager@master

Details

In response to this:

/test verify (pull_request)

Instructions for interacting with me using PR comments are available here. If you have questions or suggestions related to my behavior, please file an issue against the kubernetes-sigs/prow repository.

jianzhangbjz · 2025-12-11T07:52:52Z

Address this failure in this PR.

Run make verify
./scripts/update_codegen.sh
Generating client code for 4 targets
Generating lister code for 4 targets
I1211 07:40:00.841698    4725 main.go:58] Completed successfully.
Generating informer code for 4 targets
I1211 07:40:02.735407    4806 main.go:58] Completed successfully.
Generating openapi code for 3 targets
I1211 07:40:06.508083    5035 openapi.go:759] [github.com/operator-framework/operator-lifecycle-manager/pkg/package-server/apis/operators/v1.CSVDescription] Annotations map[string]string: tag listType on type Map; only allowed on type Slice
I1211 07:40:06.546684    5035 openapi.go:759] [k8s.io/apimachinery/pkg/apis/meta/v1.Status] Details *k8s.io/apimachinery/pkg/apis/meta/v1.StatusDetails: tag listType on type Pointer; only allowed on type Slice
I1211 07:40:06.626928    5035 api_linter.go:43] Assembling file "/tmp/update_codegen.sh.api_violations.MxNXSw"
I1211 07:40:13.553859    5453 main.go:58] Completed successfully.
I1211 07:40:15.957573    5588 main.go:58] Completed successfully.
make diff
make[1]: Entering directory '/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager'
git diff --exit-code
make[1]: Leaving directory '/home/runner/work/operator-lifecycle-manager/operator-lifecycle-manager'
# Generate mocks and silence the following warning:
# WARNING: Invoking counterfeiter multiple times from "go generate" is slow.
# Consider using counterfeiter:generate directives to speed things up.
# See https://github.com/maxbrunsfeld/counterfeiter#step-2b---add-counterfeitergenerate-directives for more information.
# Set the "COUNTERFEITER_NO_GENERATE_WARNING" environment variable to suppress this message.
# golang.org/x/tools/imports
Error: ../../../hack/overlays/goimports_vendorlesspath.go:6:6: VendorlessPath redeclared in this block
Error: 	../../../vendor/golang.org/x/tools/imports/forward.go:75:6: other declaration of VendorlessPath
pkg/api/wrappers/deployment_install_client.go:1: running "go": exit status 1
# golang.org/x/tools/imports
Error: ../../../hack/overlays/goimports_vendorlesspath.go:6:6: VendorlessPath redeclared in this block
Error: 	../../../vendor/golang.org/x/tools/imports/forward.go:75:6: other declaration of VendorlessPath
pkg/controller/bundle/bundle_unpacker.go:323: running "go": exit status 1
...
...

jianzhangbjz · 2025-12-15T09:12:21Z

/hold waiting for the downstream test pass.

jianzhangbjz · 2025-12-17T00:44:37Z

/unhold since the downstream test passed.

grokspawn · 2026-01-21T14:53:24Z

Hey @jianzhangbjz thanks for the PR! Would you consider re-titling this s/SNO/single-replica topology/? SNO is a downstream project which uses OLM(v0) but the concern can be generalized as any small-footprint deployment.

jianzhangbjz · 2026-01-23T02:10:24Z

Sure. Updated it, thanks!

camilamacedo86 · 2026-01-23T06:54:46Z

pkg/controller/operators/olm/apiservices_pod_disruption_test.go

+		// With the fix:
+		// - Single-replica + rollout + Unschedulable = expected disruption
+		// - CSV stays in current phase
+		// - ClusterOperator maintains Available=True ✅ Contract satisfied


We shoul not use emojis in the code
Can you please fix that?

Sure, I've updated it.

…se state running in single-replica topology

camilamacedo86

I am OK with
/lgtm

@perdasilva @grokspawn @tmshort
I will need to have an approval here too

@jianzhangbjz thank you a lot for raise this one.

openshift-ci · 2026-02-03T12:55:31Z

[APPROVALNOTIFIER] This PR is NOT APPROVED

This pull-request has been approved by: camilamacedo86
Once this PR has been reviewed and has the lgtm label, please assign dtfranz for approval. For more information see the Code Review Process.

The full list of commands accepted by this bot can be found here.

Details

Needs approval from an approver in each of these files:

OWNERS

Approvers can indicate their approval by writing /approve in a comment
Approvers can cancel approval by writing /approve cancel in a comment

grokspawn · 2026-02-03T21:06:32Z

pkg/controller/operators/olm/apiservices.go

+								// CRITICAL: In single-replica deployments during rollout, Unschedulable is EXPECTED
+								// due to PodAntiAffinity preventing new pod from scheduling while old pod is terminating.
+								// This is especially common in single-node clusters or control plane scenarios.
+								// Per OpenShift contract: "A component must not report Available=False during normal upgrade."


I think statements 294-296 are sufficient here and we do not need this line.

grokspawn · 2026-02-03T21:39:07Z

pkg/controller/operators/olm/apiservices_pod_disruption_test.go

+	})
+
+	t.Run("MUST NOT report Available=False during normal upgrade", func(t *testing.T) {
+		// OpenShift ClusterOperator Contract (MANDATORY):


jianzhangbjz · 2026-02-04T01:50:54Z

it looks like the preferred approach is to not run those tests in SNO environments.

Closed it due to the internal discussion.

openshift-ci bot requested review from anik120 and camilamacedo86 December 11, 2025 07:39

jianzhangbjz mentioned this pull request Dec 11, 2025

OCPBUGS-67210: Fix: Prevent Available=False during single-replica deployment upgrades in SNO openshift/operator-framework-olm#1171

Closed

jianzhangbjz force-pushed the OCPBUGS-67210 branch 2 times, most recently from 62a3848 to 1a4a9b2 Compare December 15, 2025 08:57

openshift-ci bot added the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 15, 2025

jianzhangbjz force-pushed the OCPBUGS-67210 branch 2 times, most recently from b4d66b0 to 31dc300 Compare December 16, 2025 02:38

jianzhangbjz mentioned this pull request Dec 16, 2025

🐛 Update mockgen target to remove vendorlesspath fix #3726

Merged

11 tasks

jianzhangbjz force-pushed the OCPBUGS-67210 branch from 31dc300 to c4b4fc2 Compare December 17, 2025 00:43

openshift-ci bot removed the do-not-merge/hold Indicates that a PR should not merge because someone has issued a /hold command. label Dec 17, 2025

jianzhangbjz force-pushed the OCPBUGS-67210 branch 7 times, most recently from 9082dd9 to 53d0634 Compare December 22, 2025 11:30

jianzhangbjz changed the title ~~🐛 Operator operator-lifecycle-manager-packageserver is in Available=False state running in SNO~~ 🐛 Operator operator-lifecycle-manager-packageserver is in Available=False state running in single-replica topology Jan 23, 2026

camilamacedo86 reviewed Jan 23, 2026

View reviewed changes

Operator operator-lifecycle-manager-packageserver is in Available=Fal…

582e174

…se state running in single-replica topology

jianzhangbjz force-pushed the OCPBUGS-67210 branch from 53d0634 to 582e174 Compare January 28, 2026 02:57

camilamacedo86 approved these changes Feb 3, 2026

View reviewed changes

openshift-ci bot assigned camilamacedo86 Feb 3, 2026

openshift-ci bot added the lgtm Indicates that a PR is ready to be merged. label Feb 3, 2026

grokspawn reviewed Feb 3, 2026

View reviewed changes

jianzhangbjz closed this Feb 4, 2026

Conversation

jianzhangbjz commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jianzhangbjz commented Dec 11, 2025

Uh oh!

openshift-ci bot commented Dec 11, 2025

Uh oh!

jianzhangbjz commented Dec 11, 2025

Uh oh!

openshift-ci bot commented Dec 11, 2025

Uh oh!

jianzhangbjz commented Dec 11, 2025

Uh oh!

openshift-ci bot commented Dec 11, 2025

Uh oh!

jianzhangbjz commented Dec 11, 2025 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

jianzhangbjz commented Dec 15, 2025

Uh oh!

jianzhangbjz commented Dec 17, 2025

Uh oh!

grokspawn commented Jan 21, 2026

Uh oh!

jianzhangbjz commented Jan 23, 2026

Uh oh!

camilamacedo86 Jan 23, 2026

Choose a reason for hiding this comment

Uh oh!

jianzhangbjz Jan 28, 2026

Choose a reason for hiding this comment

Uh oh!

camilamacedo86 left a comment

Choose a reason for hiding this comment

Uh oh!

openshift-ci bot commented Feb 3, 2026

Uh oh!

grokspawn Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

grokspawn Feb 3, 2026

Choose a reason for hiding this comment

Uh oh!

jianzhangbjz commented Feb 4, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

jianzhangbjz commented Dec 11, 2025 •

edited

Loading

jianzhangbjz commented Dec 11, 2025 •

edited

Loading