Skip to content

[WIP] feat(flink): Blocking RLI bootstrap when there is pending instant on job restart#18282

Open
cshuo wants to merge 2 commits intoapache:masterfrom
cshuo:rli_blocking_on_restart
Open

[WIP] feat(flink): Blocking RLI bootstrap when there is pending instant on job restart#18282
cshuo wants to merge 2 commits intoapache:masterfrom
cshuo:rli_blocking_on_restart

Conversation

@cshuo
Copy link
Collaborator

@cshuo cshuo commented Mar 5, 2026

Describe the issue this Pull Request addresses

Summary and Changelog

Impact

Risk Level

Documentation Update

Contributor's checklist

  • Read through contributor's guide
  • Enough context is provided in the sections above
  • Adequate tests were added if applicable

…cket assigner

* Support scan all record index locations
* Support RocksDb index backend.
* Add a bootstrap operator for bootstraping RLI into RocksDB index backend.
* Blocking rli bootstrap on job restart/failover until all pending instants being committed successfully.
@cshuo cshuo force-pushed the rli_blocking_on_restart branch from 7a206de to 28091e1 Compare March 5, 2026 07:53
@github-actions github-actions bot added the size:XL PR with lines of changes > 1000 label Mar 5, 2026
@cshuo cshuo force-pushed the rli_blocking_on_restart branch from 28091e1 to 95cb138 Compare March 5, 2026 10:14
@cshuo cshuo force-pushed the rli_blocking_on_restart branch from 95cb138 to 98b13da Compare March 6, 2026 10:57
@codecov-commenter
Copy link

Codecov Report

❌ Patch coverage is 61.11111% with 14 lines in your changes missing coverage. Please review.
✅ Project coverage is 57.25%. Comparing base (3dcf4c6) to head (98b13da).
⚠️ Report is 4 commits behind head on master.

Files with missing lines Patch % Lines
...pache/hudi/metadata/HoodieBackedTableMetadata.java 73.33% 5 Missing and 3 partials ⚠️
...rg/apache/hudi/common/data/HoodieListPairData.java 0.00% 3 Missing ⚠️
...va/org/apache/hudi/common/data/HoodiePairData.java 0.00% 2 Missing ⚠️
.../org/apache/hudi/metadata/HoodieTableMetadata.java 0.00% 1 Missing ⚠️
Additional details and impacted files
@@             Coverage Diff              @@
##             master   #18282      +/-   ##
============================================
+ Coverage     57.22%   57.25%   +0.02%     
- Complexity    18607    18627      +20     
============================================
  Files          1951     1955       +4     
  Lines        106765   106859      +94     
  Branches      13198    13200       +2     
============================================
+ Hits          61099    61179      +80     
- Misses        39901    39915      +14     
  Partials       5765     5765              
Flag Coverage Δ
hadoop-mr-java-client 45.33% <61.11%> (+0.09%) ⬆️
spark-java-tests 47.40% <0.00%> (+0.04%) ⬆️
spark-scala-tests 45.50% <0.00%> (-0.02%) ⬇️

Flags with carried forward coverage won't be shown. Click here to find out more.

Files with missing lines Coverage Δ
...clustering/run/strategy/JavaExecutionStrategy.java 100.00% <ø> (ø)
...un/strategy/MultipleSparkJobExecutionStrategy.java 96.07% <ø> (ø)
...gleSparkJobConsistentHashingExecutionStrategy.java 91.35% <ø> (ø)
...rg/apache/hudi/util/LazyConcatenatingIterator.java 82.14% <ø> (ø)
.../org/apache/hudi/metadata/HoodieTableMetadata.java 65.00% <0.00%> (-3.43%) ⬇️
...va/org/apache/hudi/common/data/HoodiePairData.java 0.00% <0.00%> (ø)
...rg/apache/hudi/common/data/HoodieListPairData.java 61.36% <0.00%> (-0.99%) ⬇️
...pache/hudi/metadata/HoodieBackedTableMetadata.java 79.83% <73.33%> (-0.58%) ⬇️

... and 20 files with indirect coverage changes

🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

size:XL PR with lines of changes > 1000

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants