[WIP] Feature: Support TRUNCATE TABLE for Iceberg engine

[il9ue]

[Feature] Support TRUNCATE TABLE for Iceberg Engine

Overview
As part of the Antalya release, v26.1 needs to natively support the TRUNCATE TABLE command for the Iceberg database engine. Currently, upstream ClickHouse explicitly rejects this operation. As of PR ClickHouse#91713, executing TRUNCATE down-casts to StorageObjectStorage, where it immediately throws an ErrorCodes::NOT_IMPLEMENTED exception for Data Lake engines.

To support standard analytics workflows and testing pipelines without requiring users to DROP and recreate tables (which breaks catalog bindings), implementing a metadata-only truncation is essential.

Proposed Architecture
Unlike a standard MergeTree truncation that physically drops parts from the local disk, Iceberg truncation must be entirely logical. The implementation will leave physical file garbage collection to standard Iceberg maintenance operations and focus strictly on metadata manipulation.

Core Workflow:

1. Bypass the Upstream Block: Modify StorageObjectStorage::truncate to check data_lake_metadata->supportsTruncate().
2. Snapshot Generation: Generate a new Iceberg snapshot ID and increment the metadata version (v<N+1>.metadata.json).
3. Empty Avro Manifest List: The new snapshot cannot simply omit the manifest list. We must use ClickHouse's internal object storage and Avro APIs to generate and write a strictly typed, empty Avro manifest list to object storage.
4. Metadata Update: Attach the new empty manifest list to the new snapshot, append the snapshot to the snapshots array, and update the snapshot-log and current-snapshot-id.
5. Catalog Commit: Perform an atomic swap via the ICatalog interface (e.g., REST Catalog) to point the table to the newly generated metadata JSON.

Implementation Details
The required changes span the following internal abstractions:

• src/Storages/ObjectStorage/StorageObjectStorage.h/cpp: Override truncate and remove the hardcoded throw Exception. Delegate to IDataLakeMetadata.
• src/Storages/ObjectStorage/DataLakes/IDataLakeMetadata.h: Introduce supportsTruncate() and truncate(ContextPtr, ICatalog) virtual methods.
• src/Storages/ObjectStorage/DataLakes/Iceberg/IcebergMetadata.h/cpp: Implement the core truncation logic. Must safely obtain an IObjectStorage write buffer via context->getWriteSettings() to serialize the empty Avro file before committing the JSON metadata.
• tests/integration/.../test_iceberg_truncate.py: Added Python integration tests. (Note: Stateless SQL tests are intentionally omitted as ClickHouse ENGINE=Iceberg requires an externally initialized catalog to bootstrap, which is handled via PyIceberg in the integration suite).

Acceptance Criteria

• Executing TRUNCATE TABLE ice_db.my_table succeeds without throwing NOT_IMPLEMENTED.
• SELECT count() FROM ice_db.my_table returns 0 immediately after truncation.
• A new v<N+1>.metadata.json is successfully written to the object storage warehouse.
• The REST Catalog is atomically updated to point to the new metadata version.
• Cross-Engine Compatibility: A third-party client (PyIceberg) can independently read the truncated table from the REST catalog without encountering schema or metadata corruption errors.
• CI Coverage: Integration tests successfully spin up a mock MinIO + REST Catalog, bootstrap the table, and validate the truncation logic end-to-end.

[github-actions]

Workflow [PR], commit [c81fc7f]

[arthurpassos]

I haven't implemented anything for Iceberg yet, tho it is in my todo list. I left two small comments for now.

I also looked at the transactional model and it looks ok (assuming I understood it correctly).

My understanding of the Iceberg + catalog transactional model is that updating the catalog is the commit marker, and if it fails, the transaction isn't complete even if the new metadata files were already uploaded. Those become orphan and must be ignored. This also implies an Iceberg table should always be read through a catalog if one exists, otherwise it becomes hard to determine the latest metadata snapshot.

I'll read the code more carefully later.

[arthurpassos]

Isn't isDataLake() the same as the above configuration->isDataLakeconfiguration()? see

ClickHouse/src/Storages/ObjectStorage/StorageObjectStorage.h

Line 115 in b806ac5

bool isDataLake() const override { return configuration->isDataLakeConfiguration(); }

[il9ue]

Thanks!

[arthurpassos]

I would not change this method simply to avoid merge conflicts with upstream later on

[il9ue]

@arthurpassos
just pushed a quick fix for the integration test. I had accidentally passed the S3 credentials as key-value settings to DataLakeCatalog rather than positional arguments, which caused the UNKNOWN_SETTING crash in the previous run. The Iceberg truncation test should be completely green now.

Looking at the latest CI pipeline, there are a couple of red checks remaining, but they appear to be unrelated infrastructure flakes/upstream regressions:

• aggregate_functions: Failed due to a runner CPU throttle (Code: 745... SERVER_OVERLOADED).
• test_storage_delta/test_cdf.py: Failing deep inside the Rust FFI with a delta-kernel-rs panic (Generic delta kernel error: Expected the first commit to have version 2, got None). This seems to be an existing issue with the DeltaLake CDF mock data in this branch and doesn't overlap with the Iceberg metadata changes.
• (Plus a few intermittent Docker Hub TLS handshake timeouts during environment setup).

Since the core logical truncation architecture is in place and the specific Iceberg integration tests are passing, I believe this is ready for another look whenever you have the time. Let me know if you need any further adjustments!