Use larger runners for nightly jobs#933
Closed
mhucka wants to merge 8 commits intotensorflow:masterfrom
Closed
Conversation
Make steps simpler by using tricks such as setting SHELLOPTS, and use new larger runner (from Google ML Velocity) for the build job because the job is running out of disk space on the default runner.
These are safe to use.
Make steps simpler by using tricks such as setting SHELLOPTS, and use new larger runner (from Google ML Velocity) for the build job because the job is running out of disk space on the default runner.
It's impossible to test with the new ML runners in your fork, because the fork is not part of the TensorFlow org and thus doesn't have access to the runners. Need to provide the ability to specify the runner when invoking the workflow manually.
In some contexts in GitHub Actions workflows, you have to use single quotes. Remembering when and where in YAML is too hard. Just use single quotes everywhere.s
mhucka
force-pushed
the
mh-runners-for-nightly-jobs
branch
from
7945915 to
e34f32a
Compare
mhucka
added
the
area/devops
Despite that this was valid YAML, the GitHub expression parser seems to have a hard time with string literals that contain dashes. And apparently, it's unnecessary to provide a fallback in this case anyway.
MichaelBroughton
requested changes
Dec 12, 2025
Collaborator
MichaelBroughton
left a comment
MichaelBroughton
left a comment
There was a problem hiding this comment.
This seems like it's combining a lot of changes all at once that may interact in complex ways. Can we split this into two PRs ? The first being just the runner upgrade, very simple and short. Verify that this is working well and doesn't break. Then we can move onto doing all of the other stuff:
In addition, this PR makes some simplifications to the workflows to get rid of some of the overelaborate things I did before, and make minor other changes to keep with best practices such as putting timeouts on jobs.
Member
Author
Yes, that's reasonable. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The nightly jobs are failing due to running out of disk space. (It's the same problem we had recently with the CI checks workflow.) This PR updates the jobs to use the larger self-hosted runners from Google ML Velocity.
In addition, this PR makes some simplifications to the workflows to get rid of some of the overelaborate things I did before, and make minor other changes to keep with best practices such as putting timeouts on jobs.