Skip to content

bundle deployment migrate: dotted keys in DLT pipeline configuration map cause 'key not found in map' error #4960

@antgei

Description

@antgei

Bug Report

Describe the bug

Running databricks bundle deployment migrate (with or without --noplancheck) fails when a DLT pipeline resource has Spark configuration keys containing dots (e.g. europris.swipe.egress_streaming_schema).

The Direct engine's path resolver treats dotted key names as nested map traversal paths instead of flat string keys. This causes the planning step inside CalculatePlan to fail with:

Error: cannot plan resources.pipelines.receipt_stream: cannot update configuration.europris.swipe.egress_streaming_schema with value of "${resources.schemas.egress_streaming_schema.catalog_name}...": cannot update configuration.europris.swipe.egress_streaming_schema to string: failed to navigate to parent configuration.europris.swipe: configuration.europris: key "europris" not found in map

The same config deploys correctly with the Terraform engine.

Steps to reproduce

  1. Define a DLT pipeline resource with dotted keys in the configuration block:
resources:
  pipelines:
    my_pipeline:
      configuration:
        europris.swipe.egress_streaming_schema: "${resources.schemas.my_schema.catalog_name}.${resources.schemas.my_schema.name}"
        europris.azure.subscription: ${var.AZURE.subscription}
  1. Deploy with the Terraform engine (databricks bundle deploy)
  2. Run databricks bundle deployment migrate

Expected behavior

Migration succeeds. Dotted keys in the DLT configuration map are treated as opaque flat strings, not as nested path segments.

Actual behavior

CalculatePlan attempts to traverse the dotted key as a nested map path, fails to find the intermediate node (e.g. europris), and aborts with key not found in map.

Using --noplancheck does not help — the pre-check subprocess is skipped, but the same CalculatePlan call inside migrate.go still hits the bug.

CLI version

Databricks CLI v0.296.0

Additional context

The failure path is in migrate.godeploymentBundle.CalculatePlan → Direct engine path resolution via structaccess.Set / structpath. The Terraform engine is not affected because it serializes pipeline configuration differently.

Metadata

Metadata

Assignees

Labels

BugSomething isn't workingDABsDABs related issuesengine/directSpecific to direct deployment engine in Databricks Asset Bundles

Type

No type

Projects

No projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions