Skip to content

[opt](mtmv) Optimize mv rewrite performance when mv has many partition by reduce partition mapping num#59972

Open
seawinde wants to merge 2 commits intoapache:masterfrom
seawinde:opt_init_materialization_context_slow_when_partition_much
Open

[opt](mtmv) Optimize mv rewrite performance when mv has many partition by reduce partition mapping num#59972
seawinde wants to merge 2 commits intoapache:masterfrom
seawinde:opt_init_materialization_context_slow_when_partition_much

Conversation

@seawinde
Copy link
Contributor

What problem does this PR solve?

• Summary

  • Make MTMV partition descriptor generation honor queryUsedPartitions, so only partitions referenced by the current query are carried through the pipeline instead of all partitions.
  • Applied the filter in MTMVRelatedPartitionDescOnePartitionColGenerator, preserving ordering of existing generators.
  • Added unit tests for RANGE and LIST sources to assert the filter reduces descriptors to the specified partitions.

Example
If a query only touches t1 partition p20210201, generateRelatedPartitionDescs now emits a single descriptor for p20210201 instead of all t1 partitions;

Issue Number: close #xxx

Related PR: #xxx

Problem Summary:

Release note

None

Check List (For Author)

  • Test

    • Regression test
    • Unit Test
    • Manual test (add detailed scripts or steps below)
    • No need to test or manual test. Explain why:
      • This is a refactor/code format and no logic has been changed.
      • Previous test can cover this change.
      • No code files have been changed.
      • Other reason
  • Behavior changed:

    • No.
    • Yes.
  • Does this need documentation?

    • No.
    • Yes.

Check List (For Reviewer who merge this PR)

  • Confirm the release note
  • Confirm test cases
  • Confirm document
  • Add branch pick label

@Thearas
Copy link
Contributor

Thearas commented Jan 16, 2026

Thank you for your contribution to Apache Doris.
Don't know what should be done next? See How to process your PR.

Please clearly describe your PR:

  1. What problem was fixed (it's best to include specific error reporting information). How it was fixed.
  2. Which behaviors were modified. What was the previous behavior, what is it now, why was it modified, and what possible impacts might there be.
  3. What features were added. Why was this function added?
  4. Which code was refactored and why was this part of the code refactored?
  5. Which functions were optimized and what is the difference before and after the optimization?

@seawinde
Copy link
Contributor Author

run buildall

@doris-robot
Copy link

TPC-H: Total hot run time: 31389 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpch-tools
Tpch sf100 test result on commit 06b6a3dae1420ac67e08e06b3e3d1ffe07a0e9d2, data reload: false

------ Round 1 ----------------------------------
q1	17647	4275	4046	4046
q2	2057	389	239	239
q3	10101	1286	706	706
q4	10198	810	302	302
q5	7549	2076	1802	1802
q6	187	168	136	136
q7	935	790	670	670
q8	9269	1396	1086	1086
q9	4899	4628	4611	4611
q10	6735	1801	1394	1394
q11	493	311	301	301
q12	702	739	586	586
q13	17797	3817	3103	3103
q14	299	292	278	278
q15	577	508	512	508
q16	682	672	633	633
q17	634	780	524	524
q18	6692	6418	6336	6336
q19	1103	990	623	623
q20	378	348	246	246
q21	3006	2404	2283	2283
q22	1015	1026	976	976
Total cold run time: 102955 ms
Total hot run time: 31389 ms

----- Round 2, with runtime_filter_mode=off -----
q1	4139	4047	4035	4035
q2	322	402	325	325
q3	2075	2593	2252	2252
q4	1329	1751	1326	1326
q5	4035	3960	3999	3960
q6	216	172	129	129
q7	1885	1813	1671	1671
q8	2789	2511	2396	2396
q9	7363	7194	7295	7194
q10	2744	2678	2254	2254
q11	557	463	472	463
q12	715	811	698	698
q13	3678	4102	3349	3349
q14	293	319	296	296
q15	540	503	505	503
q16	649	668	626	626
q17	1136	1370	1358	1358
q18	7883	7943	7768	7768
q19	868	919	910	910
q20	2039	2019	1939	1939
q21	4785	4441	4500	4441
q22	1127	1124	1004	1004
Total cold run time: 51167 ms
Total hot run time: 48897 ms

@doris-robot
Copy link

TPC-DS: Total hot run time: 174406 ms
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/tpcds-tools
TPC-DS sf100 test result on commit 06b6a3dae1420ac67e08e06b3e3d1ffe07a0e9d2, data reload: false

query5	4837	608	481	481
query6	323	223	233	223
query7	4245	473	269	269
query8	339	255	233	233
query9	8741	2911	2914	2911
query10	533	378	329	329
query11	15160	15069	14873	14873
query12	190	121	116	116
query13	1262	488	399	399
query14	7201	3023	2861	2861
query14_1	2668	2679	2674	2674
query15	210	194	178	178
query16	1025	491	463	463
query17	1139	650	553	553
query18	2702	428	329	329
query19	221	225	191	191
query20	125	116	122	116
query21	223	141	131	131
query22	4051	4033	4050	4033
query23	16264	15651	15568	15568
query23_1	15533	15603	15512	15512
query24	6901	1548	1191	1191
query24_1	1167	1194	1177	1177
query25	546	458	406	406
query26	1239	271	157	157
query27	2736	455	284	284
query28	4470	2186	2166	2166
query29	792	533	433	433
query30	323	248	208	208
query31	814	666	560	560
query32	93	71	73	71
query33	534	332	305	305
query34	905	888	541	541
query35	720	749	665	665
query36	859	845	828	828
query37	131	96	83	83
query38	2796	2668	2677	2668
query39	787	776	734	734
query39_1	716	714	711	711
query40	218	135	122	122
query41	72	62	61	61
query42	104	103	99	99
query43	423	477	423	423
query44	1322	758	753	753
query45	185	190	175	175
query46	822	934	579	579
query47	1443	1416	1378	1378
query48	319	324	238	238
query49	615	430	343	343
query50	612	261	195	195
query51	3718	3882	3763	3763
query52	98	107	98	98
query53	292	325	281	281
query54	292	263	254	254
query55	85	82	76	76
query56	298	292	330	292
query57	1038	1060	934	934
query58	273	263	278	263
query59	2166	2131	2065	2065
query60	334	352	315	315
query61	149	146	143	143
query62	393	350	319	319
query63	295	259	272	259
query64	4837	1255	957	957
query65	3822	3685	3714	3685
query66	1372	422	327	327
query67	15482	15591	15571	15571
query68	2368	1100	783	783
query69	447	374	322	322
query70	910	888	948	888
query71	326	304	290	290
query72	5348	3118	3196	3118
query73	596	721	320	320
query74	8782	8720	8511	8511
query75	2751	2833	2470	2470
query76	2274	1060	655	655
query77	363	368	304	304
query78	9924	9870	9128	9128
query79	1079	920	584	584
query80	1346	579	477	477
query81	575	265	232	232
query82	1344	151	109	109
query83	380	263	243	243
query84	264	113	91	91
query85	923	493	423	423
query86	405	298	291	291
query87	2874	2884	2726	2726
query88	3558	2619	2591	2591
query89	395	359	326	326
query90	1953	181	172	172
query91	169	164	143	143
query92	82	74	72	72
query93	948	925	560	560
query94	623	319	310	310
query95	590	396	317	317
query96	644	510	232	232
query97	2369	2437	2320	2320
query98	210	201	205	201
query99	625	583	499	499
Total cold run time: 248857 ms
Total hot run time: 174406 ms

@doris-robot
Copy link

ClickBench: Total hot run time: 26.79 s
machine: 'aliyun_ecs.c7a.8xlarge_32C64G'
scripts: https://github.com/apache/doris/tree/master/tools/clickbench-tools
ClickBench test result on commit 06b6a3dae1420ac67e08e06b3e3d1ffe07a0e9d2, data reload: false

query1	0.06	0.05	0.05
query2	0.10	0.04	0.04
query3	0.26	0.09	0.08
query4	1.61	0.11	0.11
query5	0.28	0.25	0.25
query6	1.15	0.67	0.64
query7	0.03	0.02	0.02
query8	0.05	0.04	0.05
query9	0.56	0.50	0.50
query10	0.54	0.56	0.55
query11	0.15	0.09	0.10
query12	0.14	0.10	0.11
query13	0.60	0.59	0.59
query14	0.96	0.95	0.95
query15	0.79	0.78	0.78
query16	0.39	0.39	0.39
query17	1.07	1.05	1.05
query18	0.23	0.22	0.21
query19	1.92	1.82	1.83
query20	0.02	0.01	0.01
query21	15.43	0.30	0.13
query22	5.42	0.05	0.05
query23	16.17	0.27	0.10
query24	1.66	0.34	0.49
query25	0.09	0.04	0.05
query26	0.15	0.14	0.13
query27	0.09	0.05	0.07
query28	3.57	1.08	0.88
query29	12.62	3.94	3.12
query30	0.28	0.14	0.12
query31	2.81	0.66	0.39
query32	3.24	0.56	0.45
query33	2.99	2.98	3.13
query34	16.31	5.05	4.48
query35	4.47	4.42	4.48
query36	0.66	0.50	0.48
query37	0.10	0.07	0.06
query38	0.07	0.04	0.04
query39	0.05	0.03	0.04
query40	0.16	0.14	0.14
query41	0.09	0.03	0.03
query42	0.04	0.04	0.03
query43	0.05	0.04	0.03
Total cold run time: 97.43 s
Total hot run time: 26.79 s

@hello-stephen
Copy link
Contributor

FE UT Coverage Report

Increment line coverage 75.00% (12/16) 🎉
Increment coverage report
Complete coverage report

@hello-stephen
Copy link
Contributor

FE Regression Coverage Report

Increment line coverage 100.00% (16/16) 🎉
Increment coverage report
Complete coverage report

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants

Comments