Skip to content

Eager agg 0115 #59944

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Draft
englefly wants to merge 26 commits into
base: master
Choose a base branch
from
+25,534 −3,181
Draft

Eager agg 0115 #59944

Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
26 commits
Select commit Hold shift + click to select a range
2badfef
eager agg from tpc_preview3 1224
englefly Dec 24, 2025
649220f
shape with/without pkfk based on tpc_preview
englefly Jan 5, 2026
5b06721
mode=1 时 即使没有经过big join 也要 强制 下推
englefly Jan 6, 2026
123f245
do not support avg/count
englefly Jan 6, 2026
d7fe0c0
group key only slotreference
englefly Jan 6, 2026
1589228
push agg on join
englefly Jan 7, 2026
f23ee82
1. remove finalGroupKeys, 2. project 下推后改写projects
englefly Jan 7, 2026
3c9685a
throw exception for eager agg when FeDebug
englefly Jan 7, 2026
fd30767
derive deep false
englefly Jan 7, 2026
937e4d0
检查context的字段 是project的输出.拒绝 sum(A) 下推 proj(x, x+y as A) 且x 不是group key
englefly Jan 8, 2026
59d57b5
simple sum-if no union
englefly Jan 8, 2026
a4b959b
sum-if 基本款 (还没有支持union), 43 有提升
englefly Jan 8, 2026
c57c90d
q5 两个sum（0）错误去重了
englefly Jan 10, 2026
026bfec
1. sum-if 不考虑穿过bigJoin， 2. 支持union
englefly Jan 13, 2026
c71df45
remove unused code
englefly Jan 13, 2026
e53f847
update shape
englefly Jan 13, 2026
5b5e2a1
DORIS-24150
englefly Jan 15, 2026
29da6a9
1. exprId 的等值判断, 2.update rt.
englefly Jan 14, 2026
b53da7f
doris-24150 rt case
englefly Jan 15, 2026
16a8580
DORIS-24151
englefly Jan 15, 2026
f5ce486
DORIS-24149
englefly Jan 15, 2026
a4174ef
aliasMap 使用HashMap，不用IdentityMap
englefly Jan 16, 2026
be1490c
DORIS-23842 没有aggFunc时 下推包含所有group key
englefly Jan 16, 2026
b57a7a0
LogicalProject 构造projectMap时不能有unbound
englefly Jan 16, 2026
61d1159
DORIS-24205
englefly Jan 16, 2026
1d069aa
DORIS-24206: fix EliminateGroupByKeyByUniform bug:没有替换alias的exprId
englefly Jan 17, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
27 changes: 13 additions & 14 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/jobs/executor/Rewriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -127,7 +127,6 @@
import org.apache.doris.nereids.rules.rewrite.PullUpProjectUnderLimit;
import org.apache.doris.nereids.rules.rewrite.PullUpProjectUnderTopN;
import org.apache.doris.nereids.rules.rewrite.PushCountIntoUnionAll;
import org.apache.doris.nereids.rules.rewrite.PushDownAggThroughJoin;
import org.apache.doris.nereids.rules.rewrite.PushDownAggThroughJoinOnPkFk;
import org.apache.doris.nereids.rules.rewrite.PushDownAggThroughJoinOneSide;
import org.apache.doris.nereids.rules.rewrite.PushDownAggWithDistinctThroughJoinOneSide;
Expand Down Expand Up @@ -171,6 +170,7 @@
import org.apache.doris.nereids.rules.rewrite.batch.ApplyToJoin;
import org.apache.doris.nereids.rules.rewrite.batch.CorrelateApplyToUnCorrelateApply;
import org.apache.doris.nereids.rules.rewrite.batch.EliminateUselessPlanUnderApply;
import org.apache.doris.nereids.rules.rewrite.eageraggregation.PushDownAggregation;
import org.apache.doris.nereids.trees.plans.algebra.SetOperation;
import org.apache.doris.nereids.trees.plans.logical.LogicalAggregate;
import org.apache.doris.nereids.trees.plans.logical.LogicalApply;
Expand Down Expand Up @@ -649,19 +649,6 @@ public class Rewriter extends AbstractBatchJobExecutor {
new MergeAggregate()
)
),
topic("Eager aggregation",
cascadesContext -> cascadesContext.rewritePlanContainsTypes(
LogicalAggregate.class, LogicalJoin.class
),
costBased(topDown(
new PushDownAggWithDistinctThroughJoinOneSide(),
new PushDownAggThroughJoinOneSide(),
new PushDownAggThroughJoin()
)),
costBased(custom(RuleType.PUSH_DOWN_DISTINCT_THROUGH_JOIN, PushDownDistinctThroughJoin::new)),
topDown(new PushCountIntoUnionAll())
),

// this rule should invoke after infer predicate and push down distinct, and before push down limit
topic("eliminate join according unique or foreign key",
cascadesContext -> cascadesContext.rewritePlanContainsTypes(LogicalJoin.class),
Expand All @@ -678,7 +665,19 @@ public class Rewriter extends AbstractBatchJobExecutor {
topDown(new PushDownAggThroughJoinOnPkFk()),
topDown(new PullUpJoinFromUnionAll())
),
topic("Eager aggregation",
cascadesContext -> cascadesContext.rewritePlanContainsTypes(
LogicalAggregate.class, LogicalJoin.class
),
costBased(topDown(
new PushDownAggWithDistinctThroughJoinOneSide(),
new PushDownAggThroughJoinOneSide()
)),

costBased(custom(RuleType.PUSH_DOWN_DISTINCT_THROUGH_JOIN, PushDownDistinctThroughJoin::new)),
custom(RuleType.PUSH_DOWN_AGG_THROUGH_JOIN, PushDownAggregation::new),
topDown(new PushCountIntoUnionAll())
),
topic("Limit optimization",
cascadesContext -> cascadesContext.rewritePlanContainsTypes(LogicalLimit.class)
|| cascadesContext.rewritePlanContainsTypes(LogicalTopN.class)
Expand Down
51 changes: 26 additions & 25 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/rules/analysis/NormalizeAggregate.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -129,33 +129,34 @@ public List<Rule> buildRules() {
.toRule(RuleType.NORMALIZE_AGGREGATE));
}

/**
* The LogicalAggregate node may contain window agg functions and usual agg functions
* we call window agg functions as window-agg and usual agg functions as trivial-agg for short
* This rule simplify LogicalAggregate node by:
* 1. Push down some exprs from old LogicalAggregate node to a new child LogicalProject Node,
* 2. create a new LogicalAggregate with normalized group by exprs and trivial-aggs
* 3. Pull up normalized old LogicalAggregate's output exprs to a new parent LogicalProject Node
* Push down exprs:
* 1. all group by exprs
* 2. child contains subquery expr in trivial-agg
* 3. child contains window expr in trivial-agg
* 4. all input slots of trivial-agg
* 5. expr(including subquery) in distinct trivial-agg
* Normalize LogicalAggregate's output.
* 1. normalize group by exprs by outputs of bottom LogicalProject
* 2. normalize trivial-aggs by outputs of bottom LogicalProject
* 3. build normalized agg outputs
* Pull up exprs:
* normalize all output exprs in old LogicalAggregate to build a parent project node, typically includes:
* 1. simple slots
* 2. aliases
* a. alias with no aggs child
* b. alias with trivial-agg child
* c. alias with window-agg
*/
@SuppressWarnings("checkstyle:UnusedLocalVariable")
private LogicalPlan normalizeAgg(LogicalAggregate<Plan> aggregate, Optional<LogicalHaving<?>> having,
public LogicalPlan normalizeAgg(LogicalAggregate<Plan> aggregate, Optional<LogicalHaving<?>> having,
CascadesContext ctx) {
// The LogicalAggregate node may contain window agg functions and usual agg functions
// we call window agg functions as window-agg and usual agg functions as trivial-agg for short
// This rule simplify LogicalAggregate node by:
// 1. Push down some exprs from old LogicalAggregate node to a new child LogicalProject Node,
// 2. create a new LogicalAggregate with normalized group by exprs and trivial-aggs
// 3. Pull up normalized old LogicalAggregate's output exprs to a new parent LogicalProject Node
// Push down exprs:
// 1. all group by exprs
// 2. child contains subquery expr in trivial-agg
// 3. child contains window expr in trivial-agg
// 4. all input slots of trivial-agg
// 5. expr(including subquery) in distinct trivial-agg
// Normalize LogicalAggregate's output.
// 1. normalize group by exprs by outputs of bottom LogicalProject
// 2. normalize trivial-aggs by outputs of bottom LogicalProject
// 3. build normalized agg outputs
// Pull up exprs:
// normalize all output exprs in old LogicalAggregate to build a parent project node, typically includes:
// 1. simple slots
// 2. aliases
// a. alias with no aggs child
// b. alias with trivial-agg child
// c. alias with window-agg

// Push down exprs:
// collect group by exprs
Set<Expression> groupingByExprs = Utils.fastToImmutableSet(aggregate.getGroupByExpressions());
Expand Down
23 changes: 23 additions & 0 deletions fe/fe-core/src/main/java/org/apache/doris/nereids/rules/rewrite/ExprIdRewriter.java
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -25,6 +25,7 @@
import org.apache.doris.nereids.rules.expression.ExpressionRewrite;
import org.apache.doris.nereids.rules.expression.ExpressionRuleExecutor;
import org.apache.doris.nereids.rules.expression.ExpressionRuleType;
import org.apache.doris.nereids.trees.expressions.Alias;
import org.apache.doris.nereids.trees.expressions.ExprId;
import org.apache.doris.nereids.trees.expressions.Expression;
import org.apache.doris.nereids.trees.expressions.Slot;
Expand Down Expand Up @@ -94,7 +95,25 @@ public Expression visitSlotReference(SlotReference slot, Map<ExprId, ExprId> rep
}
}
}

@Override
public Expression visitAlias(Alias alias, Map<ExprId, ExprId> replaceMap) {
ExprId newId = replaceMap.get(alias.getExprId());
if (newId == null) {
return alias;
}
ExprId lastId = newId;
while (true) {
newId = replaceMap.get(lastId);
if (newId == null) {
return alias.withExprId(lastId);
} else {
lastId = newId;
}
}
}
};

private final Map<ExprId, ExprId> replaceMap;

public ReplaceRule(Map<ExprId, ExprId> replaceMap) {
Expand All @@ -107,6 +126,10 @@ public List<ExpressionPatternMatcher<? extends Expression>> buildRules() {
matchesType(SlotReference.class).thenApply(ctx -> {
Slot slot = ctx.expr;
return slot.accept(SLOT_REPLACER, replaceMap);
}).toRule(ExpressionRuleType.EXPR_ID_REWRITE_REPLACE),
matchesType(Alias.class).thenApply(ctx -> {
Alias alias = ctx.expr;
return alias.accept(SLOT_REPLACER, replaceMap);
}).toRule(ExpressionRuleType.EXPR_ID_REWRITE_REPLACE)
);
}
Expand Down
Loading
Loading