Add Kueue default config for Dynamic Slicing by FIoannides · Pull Request #5693 · GoogleCloudPlatform/cluster-toolkit

FIoannides · 2026-05-20T13:24:53Z

Introduce enable_dynamic_slicing_for_tpus, adding:

the two more default Kueue configs,
Controller overrides defaults,
Validations.

gemini-code-assist · 2026-05-20T13:25:07Z

Summary of Changes

Hello, I'm Gemini Code Assist¹! I'm currently reviewing this pull request and will post my feedback shortly. In the meantime, here's a summary to help you and other reviewers quickly get up to speed!

This pull request enhances the GKE TPU workload management by integrating support for Kueue dynamic slicing. It updates the infrastructure configuration to allow users to enable dynamic slicing via new variables, introduces dedicated configuration templates, and adds robust validation logic to ensure that the required cluster settings are correctly configured for this feature.

Highlights

Dynamic Slicing Support: Added support for GKE TPU dynamic slicing in Kueue configurations, allowing for more flexible resource allocation.
Configuration Templates: Introduced new Terraform templates for Kueue configuration to support dynamic slicing, both independently and in combination with Pathways.
Validation Logic: Implemented Terraform preconditions to ensure correct configuration of accelerator topology mode, slice controller, and machine types when dynamic slicing is enabled.

Using Gemini Code Assist

The full guide for Gemini Code Assist can be found on our documentation page, here are some quick tips.

Invoking Gemini

You can request assistance from Gemini at any point by creating a comment using either /gemini <command> or @gemini-code-assist <command>. Below is a summary of the supported commands on the current page.

Feature	Command	Description
Code Review	`/gemini review`	Performs a code review for the current pull request in its current state.
Pull Request Summary	`/gemini summary`	Provides a summary of the current pull request in its current state.
Comment	@gemini-code-assist	Responds in comments when explicitly tagged, both in pull request comments and review comments.
Help	`/gemini help`	Displays a list of available commands.

Customization

To customize the Gemini Code Assist for GitHub experience, repository maintainers can create a configuration file and/or provide a custom code review style guide (such as PEP-8 for Python) by creating and adding files to a .gemini/ folder in the base of the repository. Detailed instructions can be found here.

Limitations & Feedback

Gemini Code Assist may make mistakes. Please leave feedback on any instances where its feedback is incorrect or counterproductive. You can react with 👍 and 👎 on @gemini-code-assist comments. If you're interested in giving your feedback about your experience with Gemini Code Assist for GitHub and other Google products, sign up here.

Review the Privacy Notices, Generative AI Prohibited Use Policy, Terms of Service, and learn how to configure Gemini Code Assist in GitHub here. Gemini can make mistakes, so double check it and use code with caution. ↩

gemini-code-assist

Code Review

This pull request introduces support for TPU dynamic slicing in GKE by adding new Kueue configuration templates and updating the kubectl-apply module with selection logic and validation rules. Review feedback identifies a need to broaden the machine_type validation regex to include ct types, suggests refactoring nested ternary expressions into a map lookup for improved maintainability, and recommends removing hardcoded TPU partition labels in the templates to support varied topologies.

gemini-code-assist

Code Review

This pull request introduces support for TPU Dynamic Slicing across GKE blueprints and the kubectl-apply management module, adding new Kueue configuration templates and validation logic. Feedback identifies that some blueprint settings reference non-existent module outputs, which should be resolved by defining them in the source modules to support automatic wiring. Additionally, the regex for TPU machine type validation is too restrictive and needs expansion, while hardcoded node labels in the templates should be parameterized for reusability. A redundant try() function in the Terraform configuration was also noted for removal.

jamOne- · 2026-06-01T11:10:52Z

/gemini review

gemini-code-assist

Code Review

This pull request introduces support for Dynamic Slicing for TPUs in GKE clusters by adding configuration variables, updating the Kueue installation module, and introducing new Kueue configuration templates. The review feedback highlights a few critical issues: first, the use of double dollar signs ($$) in the dynamic slicing template escapes interpolation and must be corrected to single dollar signs ($); second, the TPU partition topology label is hardcoded to 4x4x4 in both new templates, which should be made dynamic to support other topologies; and finally, the hardcoded validation check for tpu7x machine types in variables.tf should be removed to avoid unnecessary maintenance overhead.

Neelabh94 · 2026-06-01T15:55:09Z

/gcbrun

Neelabh94 · 2026-06-02T07:19:58Z

/gcbrun

Neelabh94 · 2026-06-02T08:45:10Z

/gcbrun

Neelabh94

nit comments, otherwise LGTM

Neelabh94 · 2026-06-02T12:48:56Z

/gcbrun

Neelabh94 · 2026-06-02T13:35:24Z

/gcbrun

Neelabh94

LGTM

Neelabh94 · 2026-06-03T08:45:17Z

The PR test for TPU7x also failed with the following error

Step #1 - "gke-tpu-7x": Error: module "workload-manager-install" uses module "gke-tpu-7x-pool", but matching setting and outputs were not found. This may be because the value is set explicitly or set by a prior used module
Step #1 - "gke-tpu-7x": 168:     use: [gke-tpu-7x-cluster, gke-tpu-7x-pool]
Step #1 - "gke-tpu-7x":                                    ^
Step #1 - "gke-tpu-7x": 2026-06-03T08:36:33Z: One or more blueprint validators has failed. See messages above for suggested
Step #1 - "gke-tpu-7x": actions. General troubleshooting guidance and instructions for configuring validators are shown below.
Step #1 - "gke-tpu-7x": 
Step #1 - "gke-tpu-7x": - https://goo.gle/hpc-toolkit-troubleshooting
Step #1 - "gke-tpu-7x": - https://goo.gle/hpc-toolkit-validation
Step #1 - "gke-tpu-7x": 
Step #1 - "gke-tpu-7x": Validators can be silenced or treated as warnings or errors:
Step #1 - "gke-tpu-7x": 
Step #1 - "gke-tpu-7x": - https://goo.gle/hpc-toolkit-validation-levels
Step #1 - "gke-tpu-7x": 2026-06-03T08:36:33Z: Validation failed due to the issues listed above```


Link for complete logs: https://pantheon.corp.google.com/cloud-build/builds/46acf185-373e-42bd-837d-eeac3c60ca90?project=hpc-toolkit-dev&e=-13802955

Please resolve this as well.

FIoannides · 2026-06-03T13:58:13Z

The PR test for TPU7x also failed with the following error

Step #1 - "gke-tpu-7x": Error: module "workload-manager-install" uses module "gke-tpu-7x-pool", but matching setting and outputs were not found. This may be because the value is set explicitly or set by a prior used module
Step #1 - "gke-tpu-7x": 168:     use: [gke-tpu-7x-cluster, gke-tpu-7x-pool]
Step #1 - "gke-tpu-7x":                                    ^
Step #1 - "gke-tpu-7x": 2026-06-03T08:36:33Z: One or more blueprint validators has failed. See messages above for suggested
Step #1 - "gke-tpu-7x": actions. General troubleshooting guidance and instructions for configuring validators are shown below.
Step #1 - "gke-tpu-7x": 
Step #1 - "gke-tpu-7x": - https://goo.gle/hpc-toolkit-troubleshooting
Step #1 - "gke-tpu-7x": - https://goo.gle/hpc-toolkit-validation
Step #1 - "gke-tpu-7x": 
Step #1 - "gke-tpu-7x": Validators can be silenced or treated as warnings or errors:
Step #1 - "gke-tpu-7x": 
Step #1 - "gke-tpu-7x": - https://goo.gle/hpc-toolkit-validation-levels
Step #1 - "gke-tpu-7x": 2026-06-03T08:36:33Z: Validation failed due to the issues listed above```


Link for complete logs: https://pantheon.corp.google.com/cloud-build/builds/46acf185-373e-42bd-837d-eeac3c60ca90?project=hpc-toolkit-dev&e=-13802955

Please resolve this as well.

Should be fixed now

Neelabh94 · 2026-06-03T17:35:28Z

/gcbrun

SwarnaBharathiMantena · 2026-06-04T04:37:48Z

/gcbrun

github-actions Bot added the external PR from external contributor label May 20, 2026

gemini-code-assist Bot reviewed May 20, 2026

View reviewed changes

FIoannides force-pushed the feature/dynamic-slicing-kueue branch from 8237060 to 48a7548 Compare May 20, 2026 13:34

jamOne- suggested changes May 20, 2026

View reviewed changes

FIoannides force-pushed the feature/dynamic-slicing-kueue branch 6 times, most recently from 1bbae7d to 7a8e804 Compare May 20, 2026 15:50

feat: Add Kueue default config for Dynamic Slicing

b646e94

FIoannides force-pushed the feature/dynamic-slicing-kueue branch from 7a8e804 to b646e94 Compare May 20, 2026 15:55

jamOne- approved these changes May 21, 2026

View reviewed changes

FIoannides marked this pull request as ready for review May 21, 2026 10:17

FIoannides requested a review from a team as a code owner May 21, 2026 10:17

Neelabh94 added the release-improvements Added to release notes under the "Improvements" heading. label May 26, 2026

Neelabh94 requested changes May 26, 2026

View reviewed changes

Comment thread modules/management/kubectl-apply/kueue/kueue-configuration-dynamic-slicing-pathways.yaml.tftpl Outdated

Comment thread examples/gke-tpu-7x/gke-tpu-7x.yaml

Comment thread modules/management/kubectl-apply/variables.tf

minor fixes

091f67c

FIoannides force-pushed the feature/dynamic-slicing-kueue branch from 36a45df to 091f67c Compare May 29, 2026 11:05

FIoannides requested a review from Neelabh94 May 29, 2026 11:08

gemini-code-assist Bot reviewed Jun 1, 2026

View reviewed changes

dollar sign fix

8ea7784

FIoannides force-pushed the feature/dynamic-slicing-kueue branch from 511ec7e to 8ea7784 Compare June 2, 2026 07:37

Neelabh94 previously approved these changes Jun 2, 2026

View reviewed changes

Comment thread examples/gke-tpu-7x/gke-tpu-7x-advanced.yaml Outdated

Comment thread examples/gke-tpu-7x/gke-tpu-7x.yaml Outdated

FIoannides dismissed Neelabh94’s stale review via f3a4765 June 2, 2026 12:46

FIoannides force-pushed the feature/dynamic-slicing-kueue branch from f3a4765 to 8c0c91d Compare June 2, 2026 13:33

Neelabh94 previously approved these changes Jun 2, 2026

View reviewed changes

SwarnaBharathiMantena reviewed Jun 3, 2026

View reviewed changes

Comment thread modules/management/kubectl-apply/variables.tf Outdated

Comment thread examples/gke-tpu-7x/gke-tpu-7x.yaml Outdated

FIoannides dismissed Neelabh94’s stale review via 09657a0 June 3, 2026 13:36

FIoannides force-pushed the feature/dynamic-slicing-kueue branch 3 times, most recently from 060e5d0 to 4046060 Compare June 3, 2026 13:52

FIoannides requested review from Neelabh94, SwarnaBharathiMantena and jamOne- June 3, 2026 13:55

update examples

76d9ade

FIoannides force-pushed the feature/dynamic-slicing-kueue branch from 4046060 to 76d9ade Compare June 3, 2026 18:25

Conversation

FIoannides commented May 20, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

gemini-code-assist Bot commented May 20, 2026

Summary of Changes

Highlights

Footnotes

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

jamOne- commented Jun 1, 2026

Uh oh!

gemini-code-assist Bot left a comment

Choose a reason for hiding this comment

Code Review

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Uh oh!

Neelabh94 commented Jun 1, 2026

Uh oh!

Neelabh94 commented Jun 2, 2026

Uh oh!

Neelabh94 commented Jun 2, 2026

Uh oh!

Neelabh94 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Neelabh94 commented Jun 2, 2026

Uh oh!

Neelabh94 commented Jun 2, 2026

Uh oh!

Neelabh94 left a comment

Choose a reason for hiding this comment

Uh oh!

Uh oh!

Uh oh!

Neelabh94 commented Jun 3, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

FIoannides commented Jun 3, 2026

Uh oh!

Neelabh94 commented Jun 3, 2026

FIoannides commented May 20, 2026 •

edited

Loading

Neelabh94 commented Jun 3, 2026 •

edited

Loading