refactor: Multi-Node Jobs ended_at time not being set and Complete status

[yuechao-qin]

Background

Kubernetes Job conditions use "Complete" for successful jobs and "Failed" for failed jobs. The LaunchedKubernetesJob.ended_at property was checking for "Succeeded" instead of "Complete", which meant ended_at was always None for successful K8s Jobs. This caused the API to return no end time for completed executions.

Separately, when the orchestrator hit an internal error processing a running execution (SYSTEM_ERROR), it set the status but never recorded ended_at, making it impossible to tell when the failure occurred.

Refactor

• Created enums to set the JobConditionType and ConditionStatus
• All terminal state fields (status, ended_at, exit_code, started_at) are set through a single helper to prevent future inconsistencies

Changes

cloud_pipelines_backend/launchers/kubernetes_launchers.py

• Added JobConditionType and ConditionStatus enums with K8s API doc references, replacing magic strings throughout the file
• Fixed ended_at to check ("Complete", "Failed") instead of ("Succeeded", "Failed")

cloud_pipelines_backend/orchestrator_sql.py

• Added _set_terminal_state() helper that sets all terminal fields on a ContainerExecution in one place
• Refactored SUCCEEDED, FAILED, CANCELLED, and SYSTEM_ERROR branches to use the helper
• SYSTEM_ERROR branch now sets ended_at (was previously missing)

Test

uv run pytest tests/test_kubernetes_launchers.py tests/test_orchestrator_terminal_state.py

[yuechao-qin]

• refactor: Multi-Node Jobs ended_at time not being set and Complete status #184 👈 (View in Graphite)
• fix: Launchers - Kubernetes - Jobs - Fixed LaunchedKubernetesJob.ended_at plus Set ExecutionNode.ended_at when SYSTEM_ERROR occurs in Orchestrator #200
• master

This stack of pull requests is managed by Graphite. Learn more about stacking.

[Ark-kun]

I'm not sure this function adds any value. It might also be adding confusion as it's not obvious (from the sall site) what exactly it is doing. Let's revert it.

[yuechao-qin]

I messed up on this function. I meant to make the ended_at also required, not optional. So that it's clear for any terminal state transitions it requires 1) status and 2) ended time changes. I renamed the function to be _record_terminal_state and ended_at to be required for more clarity.

Do you think this function is helpful then?

Else, do you have any suggestions on how to 1) make it clear to the developers where the terminal transitions are and 2) what required changes (status, ended_at) are needed when transitioning to a terminal state?

[Ark-kun]

Nit: This looks like a fix combined with a refactoring. This increases the diff and makes it harder to see what has changed and what hasn't. The fix changes are hidden among refactoring changes.

[Ark-kun]

Can you please extract the Job completion fix, so that we can merge it ASAP?

[yuechao-qin]

Created another branch that has the fix https://app.graphite.com/github/pr/TangleML/tangle/200
This branch is for refactoring.