HDDS-13590. Refactor HealthyPipelineSafeModeRule to not use PipelineReportFromDatanode

[priyeshkaratha]

What changes were proposed in this pull request?

Refactored HealthyPipelineSafeModeRule to make it simpler and more reliable.
Earlier, it depended on events and internal state to track pipelines. Now, it directly checks the current state from the PipelineManager whenever validation is needed.

HealthyPipelineSafeModeRule.java

• Removed event-based tracking: Eliminated processedPipelineIDs and unProcessedPipelineSet HashSets that tracked pipeline events incrementally
• Direct validation approach: Modified validate() method to directly query PipelineManager for current open pipelines instead of relying on accumulated event state
• New health check logic: Added isPipelineHealthy() helper method that validates:
  Pipeline has exactly 3 nodes
  All nodes are in healthy state (NodeStatus.inServiceHealthy())
  All nodes are registered with the node manager
• Simplified event processing: The process() method now only logs events instead of maintaining state, as validation happens directly during validate() calls
• Cleaner lifecycle management: Updated cleanup() and initializeRule() methods to remove tracking state management

SafeModeMetrics.java

• Changed currentHealthyPipelinesCount from counter (MutableCounterLong) to gauge (MutableGaugeLong)
  Replaced incCurrentHealthyPipelinesCount() with setNumCurrentHealthyPipelines(long)
• This aligns with the new approach where the value is set directly based on current state rather than incremented on events

Test Updates

• TestHealthyPipelineSafeModeRule.java: Removed assertions that relied on event processing sequence; updated expected log messages
• TestSCMSafeModeManager.java: Significant updates to handle immediate availability of healthy pipeline counts and added logic to handle edge cases where pipeline thresholds exceed available pipelines

What is the link to the Apache JIRA

HDDS-13590

How was this patch tested?

Tested using existing testcases written for safemode validation rules and Safemode Manager.
Validated using forked CI.

[sumitagrawl]

@priyeshkaratha Have few comments, logs will increase too much with change and rule once satisfied may not be satisfied next time, which is change in behavior. Check the impact.

[sumitagrawl]

In current implementation, if Rule is satisfied once, its marked exit. Never enter again. But with having dynamic rule check, one time may be success and other time may be failure.
This may not satisfy the purpose.

[sumitagrawl]

1. move logic from validate to process
2. log only one reported as event in process()

[aryangupta1998]

Thanks for the patch @priyeshkaratha. Figured out a bug that can make the SCM exit HealthyPipelineSafeModeRule early, please take a look!

[priyeshkaratha]

@sumitagrawl @aryangupta1998 Thanks for the review and I have addressed your comments.

[aryangupta1998]

Thanks for addressing the comments @priyeshkaratha, LGTM!

[adoroszlai]

@sumitagrawl would you like to take another look?

[adoroszlai]

Thanks @priyeshkaratha for the patch, @aryangupta1998, @sumitagrawl for the review.