Skip to content

Comments

DATAPLT-1268 Add shortcut for Iceberg-backed Glue Table#161

Merged
majacob merged 21 commits intomasterfrom
dataplt-1268-create_glue_iceberg
Feb 9, 2026
Merged

DATAPLT-1268 Add shortcut for Iceberg-backed Glue Table#161
majacob merged 21 commits intomasterfrom
dataplt-1268-create_glue_iceberg

Conversation

@jeffhiltz
Copy link
Contributor

@jeffhiltz jeffhiltz commented Dec 19, 2025

Adds a GlueIcebergTable shortcut for creating AWS Glue tables backed by
Apache Iceberg format. This shortcut supports:

  • Full Iceberg schema definition with complex types (list, map, struct)
  • Partition specifications for organizing data
  • Write order specifications for sorting data within files
  • Three table optimizers: retention, compaction, and orphan file deletion

Usage

const cf = require('@mapbox/cloudfriend');                                     
                                                                               
const IcebergTable = new cf.shortcuts.GlueIcebergTable({                       
  LogicalName: 'MyIcebergTable',                                               
  Name: 'my_table',                                                            
  DatabaseName: 'my_database',                                                 
  Location: 's3://my-bucket/my-table/',                                        
  Schema: {                                                                    
    Type: 'struct',                                                            
    Fields: [                                                                  
      { Id: 1, Name: 'col1', Type: 'string', Required: true },                 
      { Id: 2, Name: 'col2', Type: 'int', Required: false }                    
    ]                                                                          
  },                                                                           
  PartitionSpec: {                                                             
    SpecId: 0,                                                                 
    Fields: [{ Name: 'col1_part', SourceId: 1, Transform: 'identity' }]        
  },                                                                           
  EnableOptimizer: true,                                                       
  OptimizerRoleArn: cf.getAtt('OptimizerRole', 'Arn'),                         
  SnapshotRetentionPeriodInDays: 7                                             
});                     

Limitations

Note that CloudFormation does not expose all table optimizer configuration options. For e.g. for orphan file deletion, runRateInHours is not accepted by CFN as an adjustable parameter, at the time of this writing. As such, this shortcut supports a subset of all possible options.

Testing

Tested with mapbox internal stacks.

@jeffhiltz jeffhiltz requested a review from a team December 19, 2025 16:51
@jeffhiltz jeffhiltz added the ai AI coding agents co-authored the code label Dec 19, 2025
@majacob majacob marked this pull request as ready for review January 23, 2026 15:34
@majacob majacob requested a review from a team as a code owner January 23, 2026 15:34
@majacob majacob force-pushed the dataplt-1268-create_glue_iceberg branch from f2cf570 to 515ffe2 Compare January 27, 2026 22:58
majacob and others added 5 commits January 27, 2026 18:00
… environment values

Co-authored-by: Copilot Autofix powered by AI <62310815+github-advanced-security[bot]@users.noreply.github.com>
Copy link
Contributor Author

@jeffhiltz jeffhiltz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Just some minor comments.

@majacob majacob force-pushed the dataplt-1268-create_glue_iceberg branch from 3bb8534 to 6bfcccb Compare February 9, 2026 18:59
Copy link
Contributor Author

@jeffhiltz jeffhiltz left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

I approve of the change, but since I originally opened the PR I'm not able to approve it.

bilindhajer
bilindhajer previously approved these changes Feb 9, 2026
@majacob majacob merged commit efd6556 into master Feb 9, 2026
6 checks passed
@majacob majacob deleted the dataplt-1268-create_glue_iceberg branch February 9, 2026 21:01
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

ai AI coding agents co-authored the code

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants