gke-kueue-troubleshooting
Diagnose a GKE batch or AI workload (Job, JobSet, LeaderWorkerSet, RayJob, Kubeflow training jobs, or plain Pods) that is pending, suspended, stuck, evicted or misconfigured when Kueue manages it - from a live GKE cluster, a support bundle or exported controller/audit logs. Load it as soon as the investigation shows a Kueue footprint, even if the user never said "Kueue" - anything under kueue.x-k8s.io (labels, Pod scheduling gates, Workload/LocalQueue/ClusterQueue objects and their conditions), kueue-controller-manager events or logs, or a Job with spec.suspend=true together with a Workload ob
Try your first task
A source-grounded starter task has not been verified for this skill yet.
SKILL.md
name: gke-kueue-troubleshootingdescription: Diagnose a GKE batch or AI workload (Job, JobSet, LeaderWorkerSet, RayJob, Kubeflow training jobs, or plain Pods) that is pending, suspended, stuck, evicted or misconfigured when Kueue manages it - from a live GKE cluster, a support bundle or exported controller/audit logs. Load it as soon as the investigation shows a Kueue footprint, even if the user never said "Kueue" - anything under kueue.x-k8s.io (labels, Pod scheduling gates, Workload/LocalQueue/ClusterQueue objects and their conditions), kueue-controller-manager events or logs, or a Job with spec.suspend=true together with a Workload object in its namespace. Scripts compact cluster state and logs into verifiable facts and validate the report, while the agent classifies the cause as Kueue configuration, Kueue defect, or outside Kueue. Don't use for workloads with no Kueue footprint (use gke-node-notready, gke-workload-scaling-troubleshooting, gke-ai-troubleshooting-tpu-reservation-stockout, or gke-ai-troubleshooting-jobset-interruption).metadata: version: "1.1.0" category: ContainersGKE Kueue troubleshooting
Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, or run
kubectl patch/apply/edit/deleteor any other command that changes the cluster. When exporting logs fortrace_logs.py, run only read-only log queries scoped to the target cluster,kueue-system, Kubernetes Event logs, Admin Activity audit logs, and the incident time window as described in Log sources and View GKE logs. Give the user any fix (remediation YAML, commands, or linked documentation steps) to apply themselves in the final### Diagnostic Summary.
Usage Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read and follow the instructions in
Usage Attribution.
Prerequisites
- Tools and cluster access: Python 3.10+ with
PyYAML(all diagnostic scripts live inscripts/next to this file and require no cloud SDKs). Inlivemode, follow "Before you begin" in Deploy a batch system using Kueue to initializegcloudand select the target project, and follow Install kubectl and configure cluster access to configure cluster credentials (offline_dataandlogs_archiveread local files or stdin only). - Cloud Billing and APIs: For
livecluster inspection or Cloud Logging exports, ensure the project has an active Cloud Billing account and the Google Kubernetes Engine API (container.googleapis.com) and Cloud Logging API (logging.googleapis.com) are enabled. - Required IAM roles: Per
Kubernetes Engine IAM roles and permissions,
livecluster resource reads require Kubernetes Engine Viewer (roles/container.viewer), and reading controller pod logs viakubectl logs(container.pods.getLogs) or applying Kueue/cluster fixes requires Kubernetes Engine Developer (roles/container.developer) or Kubernetes Engine Admin (roles/container.admin). Exporting Cloud Logging and Admin Activity audit logs requires Logs Viewer (roles/logging.viewer). - Scope and related skills: Upstream Kueue ships experimental interactive
skills (
kueue-lineageandkueue-who-preempted, documented in Troubleshooting Agent Skills) for quick object-tree and preemption queries on a live cluster. This skill adds offline support-bundle and Cloud Logging compaction, GKE infrastructure boundary classification (Outside Kueue), and report validation.
1. Three-step protocol
The scripts detect the Kueue release and API version
(active_api_schema_reference._detected_kueue_version, e.g.
v0.10.x/v1beta1 vs v0.19.x/v1beta2) from controller deployments, CRDs,
object apiVersions or startup/audit logs, and load the CRD and
KueueConfiguration schema for that version. If detection_sources is only
default:v1beta2, the version was assumed — pass --kueue-version <tag>
or confirm the served CRD version before writing remediation YAML.
Where the schema comes from (schema_source / config_schema_source): in
live the cluster's served CRDs are used (live_cluster_crds, no downloads);
in offline_data bundled CRD manifests are used if present (bundled_crds);
otherwise release CRDs are read from GKE_KUEUE_TROUBLESHOOTING_KUEUE_REPO or
fetched once from GitHub (github_release:<tag>) and cached under
$XDG_CACHE_HOME/gke-kueue-troubleshooting (override with
GKE_KUEUE_TROUBLESHOOTING_CACHE_DIR; --no-remote-schema or
GKE_KUEUE_TROUBLESHOOTING_NO_REMOTE_SCHEMA=1 disables downloads).
KueueConfiguration is not a CRD; its field tree is parsed from the
KueueConfiguration v1beta2 reference
or
KueueConfiguration v1beta1 reference
matching config_api_version (falling back to builtin_defaults offline).
If schema_source is unavailable (_schema_notice present): CRD schemas
are never approximated; only KueueConfiguration may fall back to
builtin_defaults. Note in one sentence at the top of the report that the
release-exact schema was unavailable, name the remedy from _schema_notice
(cluster access, bundled CRDs, GitHub access, or --kueue-version), and
continue the diagnosis using the fields, conditions, and logs in the
snapshot (see LocalQueue
and
Topology Aware Scheduling).
Remediation YAML still uses the detected apiVersion.
Choose the data mode (--mode or GKE_KUEUE_TROUBLESHOOTING_MODE, default
live):
live: whenkubectlreaches the cluster.offline_data: directory of YAML/JSON manifests (optionally withlogs/); pass--data-path <dir>.logs_archive: exported logs only (no cluster or manifests); pass--data-path <file-or-dir>or pipe a read-only export to--data-path -(scoped per Log sources; nevercator page raw log exports into context).
Pass the same --ledger-path <ledger> to every script call; Step 3
verifies quotes and schema against it. Do not read or parse the ledger.
Output contract. snapshot.py and trace_logs.py print a digest ordered
most-important-first (detected_kueue_version, derived facts,
bulk_object_groups with counts/names/representative, individual objects/log
excerpts, schema reference last) capped at --max-output-bytes (48 KB default
for snapshot, 24 KB for logs). The complete output is written to
full_report_file (next to the ledger). If command output is truncated to a
saved file, read that saved file once instead of re-running --action get or
grep. If the digest ends with truncated:, all tagged and coverage facts are
already above it; to inspect one dropped object, use
--action get --kind <kind> --name <name> (at most 4 get calls, never the
same kind/name twice) or read a ≤ 200-line slice of full_report_file / raw
bundle files (at most twice). Never read the skill's own Python scripts.
-
Step 1 — one-shot snapshot or log timeline (schema attached)
live/offline_data: runscripts/snapshot.pywith--mode <mode> --action snapshot --ledger-path <ledger>(add--namespace <ns>if known;--kueue-namespaceif notkueue-system;--ignore-namespaces ns1,ns2for extra add-on namespaces). If no activeWorkload/Jobis present or controller/audit history is needed, also runscripts/trace_logs.py --mode <mode> --ledger-path <ledger>once (at most 2 log commands total).logs_archive: runscripts/trace_logs.py --mode logs_archive --ledger-path <ledger>with--data-path <file-or-dir>or--data-path -first (add--workload <name>to focus). Never callkubectlin this mode.- Schema lookup (optional):
scripts/snapshot.py --action explainwith--kind <Kind[.field.path]> --ledger-path <ledger>. - If a script fails: read the error once, follow
next_step, and continue with available files without retrying in a loop.
-
Step 2 — reason over the facts (the scripts do not reason for you)
- Script outputs contain facts, not conclusions
(
active_api_schema_reference,derived_graph_diagnostics,derived_log_diagnosticswithmechanism_hint,key_errors_and_events,discovered_entities,audit_resource_actions). Nothing is pre-ranked as a root cause. - Check the whole admission and execution pipeline (stages 1–6) — do not
stop at the first symptom if another independent blocker exists. For
every Workload in the request or
workload_timeline/discovered_entities, state its terminal state (Pending,Admitted,Evicted: <reason>,Requeued <n>/<limit>,Deactivated,Finished,Deleted by <owner cascade | TTL | retention policy>) and explain each terminal event. A Workload deleted via its Job'sownerReferences(user delete,ttlSecondsAfterFinished, or parent JobSet) is the Job's own lifecycle, not a Kueue retention defect — report it as why the object is gone, not as anRC<n>. Consult the Documentation catalog for the full per-stage upstream Kueue and GKE remediation links and section titles:- Object chain: Job/JobSet/LWS → Workload →
spec.queueName→LocalQueue(exists,Active) →ClusterQueue(exists,Active) →Cohort(see Troubleshooting Jobs and Troubleshooting Queues). - Quota and cohort sharing: requests per pod set × count vs
nominalQuota, borrowing (lendingLimit/borrowingLimit, cohort quota layout,reclaimWithinCohort/fairSharing),queueingStrategyordering, and GPU/TPU reservation alignment (see ClusterQueue, Cohort, and Preemption). - Flavor assignment per pod set:
ResourceFlavornodeLabels/nodeTaints/tolerations,topologyNamevs the pod set's topology request and acrossresourceGroups, andresources.transformations. Per "Implicit TAS and mixed-flavor ClusterQueues" in Topology Aware Scheduling, in aClusterQueuecombining TAS and non-TAS flavors, implicit TAS is disabled (unannotated pod sets can use only non-TAS flavors and TAS-annotated pod sets can use only TAS flavors), and a single pod set cannot mix TAS and non-TAS flavors across its requested resources. - Admission checks:
AdmissionCheckActiveand attached to theClusterQueue, DWSProvisioningRequest/ retry configuration, reservation/DWS fallback, and external controllers (see ProvisioningRequest AdmissionCheck and Run a large-scale workload with flex-start with queued provisioning). - After admission: scheduling gates (
kueue.x-k8s.io/topologyor external gates; see Troubleshooting Pods),waitForPodsReady(PodsReadyTimeout,RequeuingLimitExceeded), preemption, container/Job failure policies (backoffLimit,backoffLimitPerIndex,maxFailedIndexes,podFailurePolicy), Cluster AutoscalernoScaleUp/ stockouts, TPU slice provisioning, and GKE nodeNotReady(NetworkPluginNotReady/ CNI not initialized — follow "Troubleshoot network connectivity" in Troubleshoot nodes with the NotReady status in GKE or hand off togke-node-notready). - Controller health and configuration:
KueueConfiguration(kueue-manager-configConfigMap) fields (integrations.frameworks,manageJobsWithoutQueueName,featureGates,resources.transformations,objectRetentionPolicies), client-side rate limiting vskube-apiserveroverload (HTTP 429/503, webhook timeouts), controller restarts, and audit-logged create/delete/patch of any object in the chain.
- Object chain: Job/JobSet/LWS → Workload →
- Synthesize one unified Root Cause (
RC<n>) per underlying defect — do not fragment symptoms:- If multiple workloads fail because of the same underlying configuration
defect on a
ClusterQueueorResourceFlavor, report that single underlying defect as oneRC1rather than splitting each workload's symptom into separate root causes with conflicting fixes. - If an earlier event in the same layer is the upstream precursor of the current blocker, explain the causal chain together rather than inventing redundant competing root causes for the same failure chain.
- Conversely, when two genuinely independent defects both affect the
workload (e.g. a Kueue configuration defect and an independent external
control-plane/capacity failure, or two independent Kueue configuration
defects across different stages such as flavor/topology wiring and
AdmissionCheckattachment), report each as its ownRC<n>rather than dropping one or misclassifying a missing Kueue configuration field as aKueue defect.
- If multiple workloads fail because of the same underlying configuration
defect on a
- Defaults and uninvolved settings are not root causes — configured
mechanisms with terminal events are: a field left at its Kueue default
(such as
waitForPodsReadyomitted,waitForPodsReady.requeuingStrategy.backoffLimitCountunset/null, or defaultleaderElectiontimings) is not a root cause (RC<n>). When a workload repeatedly triggers a Kueue mechanism (such asPodsReadyTimeoutrequeuing) because its own pods or Job failure policy never reach completion or terminal failure, the single root cause (RC1) is the workload's failure/completion configuration (Outside Kueue: workload spec), while Kueue's default retry/requeue setting is background context. Do not proposewaitForPodsReady,leaderElection, orintegrations.frameworksConfigMap changes as "hardening" on quota, cohort, preemption, TAS, workload-spec, or infrastructure incidents. Conversely, a mechanism that is explicitly configured (waitForPodsReady.timeout,requeuingStrategy.backoffLimitCount,objectRetentionPolicies, preemption policies) and has a terminal event in the evidence (Evictedwith reasonPodsReadyTimeout,Deactivated/RequeuingLimitExceeded,Preempted, or deletion of deactivatedWorkloadobjects byobjectRetentionPolicies.workloads, which per "Notes" in Setup object retention policy may cascade to objects not created by Kueue such as aJobSet) is part of the causal chain and must be reported — as its ownRC<n>when the configured value causes the failure, or as an explicit stage of the chain otherwise. "Working as designed" never justifies leaving such an event unexplained. - Every root cause must be backed by a verbatim quote (from
status.conditions,spec,Events, or controller/audit logs) reproduced in#### Observations. A quota or capacity claim additionally needs a rejection fact for that same resource —insufficient quota/insufficient unused quotain a Workload condition or controller log,NotTriggerScaleUp, or aPending/FailedSchedulingreason naming the resource. AnominalQuotasmaller or larger than node allocatable,nominalQuota: 0on a queue that borrows,blockAdmission: false, or a shortttlSecondsAfterFinishedis an observation, not a root cause, unless such a rejection fact ties it to the failure. - Scope boundary — classify the causal layer (
#### Scope) before writing Root Cause or Remediation: Declare one ofKueue configuration|Kueue defect|Outside Kueue: <capacity | node | control plane | workload spec>|Mixed: <layers>|Undetermined: <what is missing>. Attribute each blocking fact by which component produced the signal and which object owns the field, not by how the symptom looks:- Locate the blocking stage(s) in the pipeline (stages 1–6):
- Stages 1–4 (queue chain, quota/cohort layout, flavor/TAS assignment,
admission check wiring) are decided by Kueue from Kueue objects
(
Kueue configuration, orKueue defectif Kueue contradicts its documented behavior for the detected version). - Stage 5 (after
Admitted=True) is decided by kube-scheduler, kubelet, node provisioner / autoscaler, admission-check controllers, and the workload's own pods. A blocker there isOutside Kueueunless a Kueue configuration caused it (e.g.ResourceFlavor.spec.nodeLabelspointing at non-existent labels, a strandedkueue.x-k8s.io/topologyscheduling gate, or cohortnominalQuotamismatched to cluster capacity). - Cohort
nominalQuota, borrowing, and Fair Sharing: Per ClusterQueue and Cohort, queues in a cohort can borrow unusednominalQuotasubject tolendingLimit(which caps how much unusednominalQuotaa member lends) andborrowingLimit. Compare cohortnominalQuotadeclarations,audit_resource_actions, and Fair Sharing / preemption settings against observed admission, preemption, or scheduling events. A queue withnominalQuota: 0is a valid borrow-only design, not a defect on its own, and#### Remediationmust modify only the resources and fields implicated by the rejection facts. Per "ClusterQueue share value" in Preemption, Kueue summarizes borrowed usage in.status.fairSharing.weightedShareweighted by.spec.fairSharing.weight(higher weight lowers share value for the same borrowed usage; Kueue admits lowest share first and preempts highest share first; when cohort members configure different.spec.fairSharing.weightvalues, report each member's weight and how Fair Sharing weights their borrowed usage in#### Observations). - Stage 6 (controller health) splits by signal provenance: facts
tagged
source side: kueue-clientreflect client-side rate limiting controlled byclientConnection.qpsandclientConnection.burstin the "ClientConnection" section of theKueueConfigurationreference; facts taggedsource side: apiserver(HTTP 429/503, API Priority and Fairness, webhook/etcd timeouts) belong toOutside Kueue: control plane(see Monitor GKE control plane metrics).
- Stages 1–4 (queue chain, quota/cohort layout, flavor/TAS assignment,
admission check wiring) are decided by Kueue from Kueue objects
(
- Independent layers →
Mixed. If both a Kueue-side blocker and an external-layer blocker independently affect the workload(s), set#### ScopetoMixed: <layers>, give each layer its ownRC<n>, and include the literal phraseOutside Kueuein the externalRC<n>header line (e.g.- RC2 (Outside Kueue — control plane): ...). Never writeOutside Kueuein a Kueue-configuration RC header or its remediation. - Never patch across layers. An
Outside KueueRC is remediated on the component that owns it (node pool, image, control plane, cloud capacity, the Job's own spec). AKueue configurationRC is remediated on the Kueue object that owns the field.
- Locate the blocking stage(s) in the pipeline (stages 1–6):
- When evidence is incomplete, use
#### Open Questions(OQ<n>):BUNDLE_COVERAGEandLOG_COVERAGEstate which object kinds and log sources are present and absent. If the runtime status objects and logs needed to confirm why a stage failed are absent from both (e.g. a static manifest-only bundle or a logs-only archive lacking manifests), do not assert an unverified guess as a definiteRC<n>: recordOQ1under#### Open Questionsstating what is missing, the candidate explanations, the discriminating observation, the read-onlyCollect:command, and the conditional remediation for each candidate. - Commit when the evidence decides — do not hedge a decided finding. When
runtime evidence (conditions, events, or logs) settles the cause, write a
definite
RC<n>under#### Root Causewith one concrete fix under#### Remediation, without listing "or possibly ..." alternatives next to it. ReserveOQ<n>for questions the available evidence genuinely cannot answer and that name one read-only command to answer them. If the first-listed candidate is the only one with a rejection or terminal fact in the evidence, it is theRC<n>, not anOQ<n>. Explicitly configured mechanisms with terminal events belong in the causal chain, not under#### Ruled Out.
- Script outputs contain facts, not conclusions
(
-
Step 3 — evidence and schema validation
- Number confirmed root causes
RC1,RC2, … and give a matching remediation entry for eachRC<n>under#### Remediation(conditional fixes for unresolved candidates go with theirOQ<n>):- For a Kueue configuration RC: link the official Kueue or GKE
documentation section from
Documentation catalog alongside the exact
kubectlcommand or YAML manifest in the detected API version, preserving allactive_api_schema_referenceinvariants and covering every resource requested by the workload's PodSets.KueueConfigurationis not an API resource: tell the user to follow "Install a custom-configured released version" and "Change the feature gates configuration" in Installation (or the GKE sections in Documentation catalog) to editdata.controller_manager_config.yaml(kind: Configuration) in thekueue-manager-configConfigMap and restart thekueue-controller-managerDeployment, showing the completecontroller_manager_config.yamldocument rather than a partial patch. - For a Kueue defect RC: do not invent unsupported workarounds that
break workload semantics. Tell the user to collect snapshot and
controller log evidence, check or file an upstream issue at
https://github.com/kubernetes-sigs/kueue/issuesor contact Cloud Customer Care per Getting support for GKE, and state the confirmation signal (Confirm by checking that ...). - For an Outside Kueue RC: never propose a
ClusterQueue,ResourceFlavor, orKueueConfigurationpatch to mask an external infrastructure failure. State the external-layer remediation (linking the relevant GKE guide from Documentation catalog or handing off togke-node-notready,gke-workload-scaling-troubleshooting,gke-ai-troubleshooting-tpu-reservation-stockout, orgke-ai-troubleshooting-jobset-interruption), state the confirmation signal (Confirm by checking that <object condition> transitions to <expected state>), and for persistent control-plane or hardware issues advise contacting Cloud Customer Care per Getting support for GKE.
- For a Kueue configuration RC: link the official Kueue or GKE
documentation section from
Documentation catalog alongside the exact
- In
#### Observations, wrap a multi-word phrase in backticks only when it is a verbatim substring of tool output (synthesised summaries go in plain text). - Save the draft and run
scripts/validate_report.pywith--draft-file <draft.md> --ledger-path <ledger>. It checks required headers,fabricated_quotes,remediation_gaps,config_remediation_errors,unknown_kubectl_resources, andschema_warnings. If it reports an error, fix it and re-run once.unsupported_root_causesis advisory: tie any listedRC<n>to a rejection/terminal signal in#### Observationsor move it to#### Open Questions; do not re-run the validator for it. - Finish with:
- Number confirmed root causes
### Diagnostic Summary
#### Scope
<Kueue configuration | Kueue defect | Outside Kueue: <layer> | Mixed>
#### Observations
- Verbatim evidence from `status.conditions`, `spec`, `Events`, or
controller/audit logs, with timestamps for historical facts.
#### Root Cause
- RC1: <precise root cause and causal mechanism>, tied to the evidence above.
- RC2 (Outside Kueue — <layer>): <if Mixed / external layer, state the external
root cause>.
#### Open Questions
- none (or `OQ1` with missing data, candidates, discriminator, read-only
`Collect:` command, and conditional remediation when data is absent).
#### Ruled Out
- Brief list of secondary observations, resolved/precursor events, or
non-blocking settings ruled out as root causes.
#### Remediation
- RC1: link the official documentation section describing the fix (plus the
exact `kubectl` command or YAML manifest with that link for Kueue RCs), or
external-layer remediation + confirmation signal (for external RCs).
- RC2: ...