HIVE-27328: Acid dirCache is not invalidated in TezAMs while dropping table by abstractdog · Pull Request #6309 · apache/hive

abstractdog · 2026-02-09T16:25:41Z

What changes were proposed in this pull request?

See jira for initial analysis.
The patch introduces a table createTime check while using the acid dir cache. This create_time property is propagated to the TezAM in the vertex-level JobConf, making it able to invalidate stale entries that belong to the previous instance of the same table (before DROP).
An RPC call-based solution wouldn't work because HS2 has no such interface to the AMs, and introducing such functionality to the TezClient/DagClient would be an epic hack, so it's not an option.

Why are the changes needed?

Because stale cache can cause problems, that are hard to investigate and that weren't taken care of by HIVE-26060.

Does this PR introduce any user-facing change?

No.

How was this patch tested?

Manually tested with minihs2.

mvn clean install -Dtest=StartMiniHS2Cluster -DminiHS2.clusterType=llap -DminiHS2.run=true -DminiHS2.usePortsFromConf=true -T 1C -Denforcer.skip=true -pl itests/hive-unit -pl itests/util -Pitests -nsu DminiHS2.isMetastoreRemote=true

set hive.explain.user=false;
set hive.query.results.cache.enabled=false;
set hive.fetch.task.conversion=none;

set hive.support.concurrency=true;
set hive.exec.dynamic.partition.mode=nonstrict;
set hive.txn.manager=org.apache.hadoop.hive.ql.lockmgr.DbTxnManager;

set hive.exec.orc.split.strategy=BI;


CREATE TABLE test_part(id int) PARTITIONED BY(dt string) CLUSTERED BY (id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES ('transactional'='true');
INSERT OVERWRITE TABLE test_part PARTITION (dt) SELECT 1, '1';

SELECT * FROM test_part;

DROP TABLE test_part;


CREATE TABLE test_part(id int) PARTITIONED BY(dt string) CLUSTERED BY (id) INTO 2 BUCKETS STORED AS ORC TBLPROPERTIES ('transactional'='true');
INSERT OVERWRITE TABLE test_part PARTITION (dt) SELECT 1, '1';

SELECT * FROM test_part;

looking for log entry in the AM log:

find . -name "syslog" | xargs grep "invalidating entry"
...
2026-02-09T08:12:00,509 INFO [ORC_GET_SPLITS #1] io.AcidUtils: Table default.test_part was recreated (at: 1770653502) since it was stored in acid cache (at: 1770653453), invalidating entry

abstractdog · 2026-02-10T08:03:53Z

oh, this causes a lot of qtest noise because of the create_time value added to TableDesc properties, need to think about this
I think I can put it directly to the MapWork's conf somehow to make it consumed in the split generation process

abstractdog · 2026-02-10T12:36:13Z

oh, this causes a lot of qtest noise because of the create_time value added to TableDesc properties, need to think about this I think I can put it directly to the MapWork's conf somehow to make it consumed in the split generation process

UPDATE: solved from MapWork

ayushtkn

Thanx @abstractdog for chasing this, dropped minor comments, rest looks good

ayushtkn · 2026-02-13T10:08:05Z

ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java

+   * @param tableCreateTime
+   *     table creation time to store, represented as a string
+   */
+  public static void setTableCreateTime(Configuration conf, Table table) {


this param doesn't exist

ayushtkn · 2026-02-13T10:09:32Z

ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java

+  public static void setTableCreateTime(Configuration conf, Table table) {
+    Objects.requireNonNull(table, "Cannot get table create time. Table object is expected to be non-null.");
+    String tableCreateTime = String.valueOf(table.getCreateTime());
+    String fullTableName = String.format("%s.%s", table.getDbName(), table.getTableName());


Can we use:

table.getFullyQualifiedName();

or

TableName.getDbTable(table.getDbName(), table.getTableName());

ayushtkn · 2026-02-13T10:11:24Z

ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java

+    Objects.requireNonNull(table, "Cannot get table create time. Table object is expected to be non-null.");
+    String tableCreateTime = String.valueOf(table.getCreateTime());
+    String fullTableName = String.format("%s.%s", table.getDbName(), table.getTableName());
+    conf.set(String.format("%s.%s", fullTableName, CREATE_TIME), tableCreateTime);


We could do setInt, rather than converting it into String and then setting

conf.setInt(String.format("%s.%s", fullTableName, CREATE_TIME), table.getCreateTime());

ayushtkn · 2026-02-13T10:17:28Z

ql/src/java/org/apache/hadoop/hive/ql/exec/Utilities.java

+  public static int getTableCreateTime(Configuration conf, String tableName) {
+    String createTime = conf.get(String.format("%s.%s", tableName, CREATE_TIME));
+    return createTime == null ? 0 : Integer.parseInt(createTime);


use

conf.getInt(...)

ayushtkn · 2026-02-13T10:19:38Z

ql/src/java/org/apache/hadoop/hive/ql/io/AcidUtils.java


+    // Check whether the table was re-created after being stored in the cache.
+    // The value null check avoids a noisy log message during the initial lookup, when no cache entry exists.
+    if (value != null && tableCreateTimeInCache < tableCreateTime) {


The value is in int. Doubting about the Interger Overflow case, Should we just do

tableCreateTimeInCache != tableCreateTime

ayushtkn · 2026-02-13T10:24:02Z

ql/src/test/org/apache/hadoop/hive/ql/plan/TestMapWork.java

+    mapWork.configureJobConf(jobConf);
+
+    // Then the table's create time should be present in the JobConf
+    String fullTableName = dbName + "." + tableName;


String fullTableName = TableName.getDbTable(dbName, tableName);

ayushtkn · 2026-02-13T10:24:59Z

ql/src/java/org/apache/hadoop/hive/ql/exec/TableScanOperator.java


+  @Override
+  public void configureJobConf(JobConf job) {
+    Table table = getConf().getTableMetadata();


can getConf() be null?, maybe we can do, if there is such a chance

public void configureJobConf(JobConf job) { if (getConf() != null && getConf().getTableMetadata() != null) { Utilities.setTableCreateTime(job, getConf().getTableMetadata()); }

… table

abstractdog · 2026-02-17T11:15:12Z

thanks @ayushtkn for your comments, all made sense, fixed them!

sonarqubecloud · 2026-02-17T15:13:50Z

Quality Gate passed

Issues
1 New issue
0 Accepted issues

Measures
0 Security Hotspots
0.0% Coverage on New Code
0.2% Duplication on New Code

See analysis details on SonarQube Cloud

ayushtkn

LGTM

abstractdog changed the title ~~HIVE-27328: Acid dirCache is not invalidated in TezAMs while dropping…~~ HIVE-27328: Acid dirCache is not invalidated in TezAMs while dropping table Feb 9, 2026

asf-ci-hive added the tests pending label Feb 9, 2026

abstractdog force-pushed the HIVE-27328 branch from c371ea4 to 9cc8925 Compare February 9, 2026 16:43

asf-ci-hive added tests failed tests pending tests unstable and removed tests pending tests failed labels Feb 9, 2026

abstractdog force-pushed the HIVE-27328 branch from 9cc8925 to 38c5e14 Compare February 10, 2026 11:30

asf-ci-hive added tests pending and removed tests unstable labels Feb 10, 2026

asf-ci-hive added tests unstable tests pending and removed tests pending tests unstable labels Feb 10, 2026

abstractdog force-pushed the HIVE-27328 branch from 80bf84e to 4cc0c07 Compare February 11, 2026 09:53

asf-ci-hive added tests pending tests passed and removed tests unstable tests pending labels Feb 11, 2026

ayushtkn reviewed Feb 13, 2026

View reviewed changes

abstractdog added 2 commits February 17, 2026 10:57

HIVE-27328: Acid dirCache is not invalidated in TezAMs while dropping…

c7510a7

… table

PR comments

ea68d9d

abstractdog force-pushed the HIVE-27328 branch from 4cc0c07 to ea68d9d Compare February 17, 2026 11:14

asf-ci-hive removed the tests passed label Feb 17, 2026

asf-ci-hive added the tests pending label Feb 17, 2026

abstractdog requested a review from ayushtkn February 17, 2026 11:15

asf-ci-hive added tests unstable tests pending and removed tests pending tests unstable labels Feb 17, 2026

asf-ci-hive added tests passed and removed tests pending labels Feb 17, 2026

ayushtkn approved these changes Feb 18, 2026

View reviewed changes

abstractdog merged commit 7060d94 into apache:master Feb 18, 2026
4 checks passed

Conversation

abstractdog commented Feb 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

What changes were proposed in this pull request?

Why are the changes needed?

Does this PR introduce any user-facing change?

How was this patch tested?

Uh oh!

abstractdog commented Feb 10, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

abstractdog commented Feb 10, 2026

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

ayushtkn Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

ayushtkn Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

ayushtkn Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

ayushtkn Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

ayushtkn Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

ayushtkn Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

ayushtkn Feb 13, 2026

Choose a reason for hiding this comment

Uh oh!

abstractdog commented Feb 17, 2026

Uh oh!

sonarqubecloud bot commented Feb 17, 2026

Quality Gate passed

Uh oh!

ayushtkn left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants

Comments

abstractdog commented Feb 9, 2026 •

edited

Loading

abstractdog commented Feb 10, 2026 •

edited

Loading