Skip to content

Conversation

@morningman
Copy link
Contributor

@morningman morningman commented Jan 16, 2026

What problem does this PR solve?

Related PR: #58166

Problem Summary:

When Observer FE receives edit logs from Master FE after a Hive insert operation, it may miss adding new partitions to its partition values cache. This happens because:

  1. When inserting into a new partition, Master FE refreshes its cache and gets the partition into its partition values cache
  2. Master FE records the partition as MODIFIED (not NEW) in the edit log if it already exists in HMS
  3. Observer FE processes only newPartNames when updating partition values cache, missing partitions marked as MODIFIED

Fix by merging both modifiedPartNames and newPartNames before adding to the partition values cache, ensuring Observer FE stays synchronized with Master FE.

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

…ng insert

When Observer FE receives edit logs from Master FE after a Hive insert operation,
it may miss adding new partitions to its partition values cache. This happens because:

1. When inserting into a new partition, Master FE refreshes its cache and gets
 the partition into its partition values cache
2. Master FE records the partition as MODIFIED (not NEW) in the edit log if it
 already exists in HMS
3. Observer FE processes only newPartNames when updating partition values cache,
 missing partitions marked as MODIFIED

Fix by merging both modifiedPartNames and newPartNames before adding to the
partition values cache, ensuring Observer FE stays synchronized with Master FE.
@hello-stephen
Copy link
Contributor

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@github-actions
Copy link
Contributor

PR approved by anyone and no changes requested.

@morningman
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 32205 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 8c4c8fcf1bc05aea34863acca7b0ce6ceb5a0e5f, data reload: false

------ Round 1 ----------------------------------
q1	17715	4269	4132	4132
q2	2251	355	237	237
q3	10230	1288	747	747
q4	10243	852	320	320
q5	7516	2130	1889	1889
q6	195	171	139	139
q7	922	803	668	668
q8	9313	1461	1286	1286
q9	4884	4596	4697	4596
q10	7240	1785	1403	1403
q11	646	321	299	299
q12	730	741	618	618
q13	17784	3850	3101	3101
q14	290	300	293	293
q15	605	515	506	506
q16	708	701	646	646
q17	738	845	516	516
q18	6775	6553	6396	6396
q19	1339	1028	681	681
q20	436	399	263	263
q21	3256	2651	2468	2468
q22	1116	1066	1001	1001
Total cold run time: 104932 ms
Total hot run time: 32205 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4173	4076	4121	4076
q2	347	419	333	333
q3	2075	2602	2243	2243
q4	1360	1810	1318	1318
q5	4135	3967	4062	3967
q6	217	175	133	133
q7	2148	1963	1729	1729
q8	2579	2491	2399	2399
q9	7237	7238	7245	7238
q10	2591	2739	2357	2357
q11	564	488	455	455
q12	734	777	651	651
q13	3765	4101	3432	3432
q14	302	318	301	301
q15	557	517	498	498
q16	672	679	638	638
q17	1180	1314	1324	1314
q18	8040	7842	7928	7842
q19	953	996	922	922
q20	2053	2208	1959	1959
q21	4970	4536	4138	4138
q22	1104	1039	981	981
Total cold run time: 51756 ms
Total hot run time: 48924 ms

@github-actions
Copy link
Contributor

PR approved by at least one committer and no changes requested.

@github-actions github-actions bot added the approved Indicates a PR has been approved by one committer. label Jan 17, 2026
@doris-robot
Copy link

TPC-DS: Total hot run time: 177343 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 8c4c8fcf1bc05aea34863acca7b0ce6ceb5a0e5f, data reload: false

query5	4897	651	503	503
query6	337	248	225	225
query7	4232	488	292	292
query8	380	254	244	244
query9	8704	3195	3175	3175
query10	491	399	350	350
query11	15326	15185	14935	14935
query12	186	120	122	120
query13	1265	516	436	436
query14	6372	3114	2904	2904
query14_1	2801	2792	2796	2792
query15	217	205	181	181
query16	3268	509	479	479
query17	1069	689	569	569
query18	2228	459	349	349
query19	232	232	211	211
query20	132	123	122	122
query21	512	145	124	124
query22	4197	4281	4108	4108
query23	16218	15650	15323	15323
query23_1	15711	15463	15401	15401
query24	7229	1577	1227	1227
query24_1	1211	1219	1239	1219
query25	688	445	403	403
query26	1347	278	152	152
query27	2603	468	293	293
query28	4532	2374	2356	2356
query29	737	518	417	417
query30	472	252	212	212
query31	875	639	576	576
query32	91	78	80	78
query33	697	362	309	309
query34	919	911	559	559
query35	740	768	701	701
query36	914	929	891	891
query37	228	103	82	82
query38	3234	2744	2718	2718
query39	774	775	741	741
query39_1	705	719	728	719
query40	370	138	123	123
query41	69	71	64	64
query42	107	103	104	103
query43	493	483	488	483
query44	1505	818	810	810
query45	187	187	183	183
query46	876	972	605	605
query47	1499	1503	1365	1365
query48	353	357	259	259
query49	686	433	362	362
query50	670	293	225	225
query51	3904	3776	3822	3776
query52	144	112	101	101
query53	306	325	278	278
query54	450	269	264	264
query55	84	82	85	82
query56	310	321	318	318
query57	1076	1023	923	923
query58	277	267	272	267
query59	2343	2273	2285	2273
query60	396	362	315	315
query61	167	164	157	157
query62	419	377	325	325
query63	310	267	273	267
query64	4528	1303	1004	1004
query65	3873	3726	3787	3726
query66	1364	442	320	320
query67	15574	15644	15502	15502
query68	2461	1169	846	846
query69	458	381	335	335
query70	1085	1024	986	986
query71	344	325	299	299
query72	5309	3133	3398	3133
query73	628	702	345	345
query74	8793	8917	8602	8602
query75	2796	2858	2496	2496
query76	1844	1084	740	740
query77	423	407	342	342
query78	9820	9916	9171	9171
query79	2881	950	625	625
query80	1689	579	505	505
query81	574	267	236	236
query82	1002	156	115	115
query83	355	277	268	268
query84	279	131	104	104
query85	939	509	445	445
query86	417	342	316	316
query87	2917	2894	2867	2867
query88	3939	2847	2814	2814
query89	410	366	341	341
query90	1941	188	172	172
query91	179	171	151	151
query92	94	75	75	75
query93	1203	993	598	598
query94	671	315	287	287
query95	598	345	335	335
query96	727	538	247	247
query97	2342	2420	2331	2331
query98	241	210	206	206
query99	635	594	545	545
Total cold run time: 256318 ms
Total hot run time: 177343 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 27.45 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 8c4c8fcf1bc05aea34863acca7b0ce6ceb5a0e5f, data reload: false

query1	0.05	0.05	0.05
query2	0.15	0.05	0.04
query3	0.28	0.09	0.08
query4	1.62	0.12	0.11
query5	0.27	0.25	0.25
query6	1.16	0.68	0.66
query7	0.03	0.02	0.03
query8	0.06	0.04	0.04
query9	0.58	0.50	0.50
query10	0.55	0.55	0.55
query11	0.17	0.11	0.11
query12	0.16	0.11	0.11
query13	0.62	0.60	0.59
query14	0.95	0.96	0.97
query15	0.82	0.79	0.78
query16	0.40	0.41	0.41
query17	1.08	1.10	1.07
query18	0.24	0.22	0.22
query19	2.07	1.93	1.94
query20	0.02	0.02	0.02
query21	15.44	0.28	0.15
query22	5.04	0.07	0.05
query23	15.72	0.29	0.10
query24	1.85	0.62	0.33
query25	0.09	0.09	0.09
query26	0.15	0.14	0.14
query27	0.08	0.05	0.06
query28	3.33	1.10	0.89
query29	12.60	4.11	3.32
query30	0.28	0.14	0.12
query31	2.83	0.67	0.41
query32	3.26	0.57	0.47
query33	3.09	3.00	3.07
query34	16.24	5.07	4.51
query35	4.44	4.54	4.47
query36	0.67	0.51	0.49
query37	0.13	0.07	0.07
query38	0.07	0.05	0.04
query39	0.07	0.03	0.04
query40	0.18	0.15	0.12
query41	0.10	0.04	0.03
query42	0.06	0.03	0.03
query43	0.04	0.04	0.04
Total cold run time: 97.04 s
Total hot run time: 27.45 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 0.00% (0/4) 🎉
Increment coverage report
Complete coverage report

@morningman morningman merged commit d321977 into apache:master Jan 17, 2026
36 of 37 checks passed
github-actions bot pushed a commit that referenced this pull request Jan 17, 2026
…ng insert (#59965)

### What problem does this PR solve?

Related PR: #58166

Problem Summary:

When Observer FE receives edit logs from Master FE after a Hive insert
operation, it may miss adding new partitions to its partition values
cache. This happens because:

1. When inserting into a new partition, Master FE refreshes its cache
and gets the partition into its partition values cache
2. Master FE records the partition as MODIFIED (not NEW) in the edit log
if it already exists in HMS
3. Observer FE processes only newPartNames when updating partition
values cache, missing partitions marked as MODIFIED

Fix by merging both modifiedPartNames and newPartNames before adding to
the partition values cache, ensuring Observer FE stays synchronized with
Master FE.
yiguolei pushed a commit that referenced this pull request Jan 18, 2026
…rver FE during insert #59965 (#59994)

Cherry-picked from #59965

Co-authored-by: Mingyu Chen (Rayner) <morningman@163.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

approved Indicates a PR has been approved by one committer. dev/3.1.x dev/4.0.3-merged reviewed

Projects

None yet

Development

Successfully merging this pull request may close these issues.

6 participants