gke-unprotected-statefulset
Audits, detects, and proposes fixes for GKE StatefulSets that lack a matching PodDisruptionBudget (PDB) to prevent simultaneous Pod evictions during voluntary node drains (GKE insight subtype PDB_UNPROTECTED_STATEFULSET). Use when investigating the PDB_UNPROTECTED_STATEFULSET insight, checking whether GKE StatefulSets have a matching PDB, or proposing fixes for selector mismatches where an existing PDB does not match the StatefulSet's .spec.selector. Don't use for existing PDBs that allow zero disruptions or block node drains (PDB_UNPERMISSIVE), general probe and PDB setup unrelated to this in
Try your first task
A source-grounded starter task has not been verified for this skill yet.
SKILL.md
name: gke-unprotected-statefulsetmetadata: version: "1.0.0" category: Containersdescription: >- Audits, detects, and proposes fixes for GKE StatefulSets that lack a matching PodDisruptionBudget (PDB) to prevent simultaneous Pod evictions during voluntary node drains (GKE insight subtype PDB_UNPROTECTED_STATEFULSET). Use when investigating the PDB_UNPROTECTED_STATEFULSET insight, checking whether GKE StatefulSets have a matching PDB, or proposing fixes for selector mismatches where an existing PDB does not match the StatefulSet's .spec.selector. Don't use for existing PDBs that allow zero disruptions or block node drains (PDB_UNPERMISSIVE), general probe and PDB setup unrelated to this insight (use gke-reliability), upgrade planning (use gke-upgrades), Pods stuck in Pending (use gke-workload-troubleshooting), node autoscaling or capacity stockouts (use gke-cluster-autoscaler), ComputeClass configuration (use gke-compute-classes), or general Kubernetes manifest generation (use gke-manifest-generation).GKE unprotected StatefulSet skill
Use this skill to find, verify, and propose fixes for GKE StatefulSet
workloads lacking a matching PodDisruptionBudget (PDB), subtype
PDB_UNPROTECTED_STATEFULSET. Without a matching PDB, voluntary node drains can
take down all Pods at once or drop a stateful app below quorum; a zero-budget
PDB triggers PDB_UNPERMISSIVE, which GKE violates after one hour (see
When GKE identifies workloads with vulnerability to disruption
and "Dealing with disruptions" and "Pod disruption budgets" in
Disruptions).
Read-only rule: Run read-only diagnostic commands only. Never drain, delete, or re-create nodes, create or modify Kubernetes resources, dismiss or change insight or recommendation state, or run any other command that changes the cluster or project. Give the user any fix to apply themselves.
Usage attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, read and
follow Usage Attribution.
Prerequisites
- Google Cloud CLI, APIs, and billing: If
gcloudis missing or uninitialized orcontainer.googleapis.comis disabled, point the user to Before you begin (never enable APIs or install software yourself). Ifrecommender.googleapis.comis disabled, point the user to "gcloud" in Enable the API. The project needs an active Cloud Billing account. kubectland cluster credentials: Ifkubectlor required auth plugins are missing, point the user to "Install kubectl" and "Install required plugins" in Install kubectl and configure cluster access, and use "Store cluster information for kubectl" on that page to configure cluster credentials forkubectl.- IAM roles: Under
Google Kubernetes Engine roles
and
Required roles:
- Kubernetes Engine Viewer (
roles/container.viewer) to configure credentials and read clusters, StatefulSets, PDBs, insights, and recommendations. - Kubernetes Engine Developer (
roles/container.developer) only for the user to create/update PDBs, update/scale StatefulSets, or dismiss recommendations. - Kubernetes Engine Cluster Admin (
roles/container.clusterAdmin) only for the user to configure cluster maintenance windows/exclusions or Standard node-pool maintenance exclusions.
- Kubernetes Engine Viewer (
- Documentation:
- Ensure workloads are disruption-ready (Sections: "Identify workloads with vulnerability to disruption", "When GKE identifies workloads with vulnerability to disruption", "Implement the guidance to improve disruption readiness")
- Optimize your usage of GKE with insights and recommendations (Sections: "Before you begin", "Required roles", "View insights and recommendations", "Dismiss or restore recommendations", "Dismiss recommendations using the Google Cloud CLI")
- Enable the API (Sections: "gcloud")
- Insights (Sections: "Description", "Content", "Target Resources", "State Info", "Severity", "Last Refresh Time")
- Google Kubernetes Engine roles and permissions (Sections: "Google Kubernetes Engine roles")
- Install kubectl and configure cluster access (Sections: "Install kubectl", "Install required plugins", "Store cluster information for kubectl")
- Manage clusters (Sections: "View your clusters")
- Deploying a stateful application (Sections: "Creating a StatefulSet", "Inspecting a StatefulSet", "Updating a StatefulSet", "Scaling a StatefulSet")
- Node upgrade strategies (Sections: "maxSurge: GKE creates a new surge node before removing an existing one", "maxUnavailable: GKE makes an existing node unavailable to recreate it", "Phase 3: Drain blue pool", "Phase 5: Delete blue pool")
- Autopilot cluster upgrades (Sections: "Automatic node upgrades")
- Troubleshoot cluster upgrades (Sections: "Troubleshoot node pool upgrades taking longer than usual")
- Maintenance windows and exclusions (Sections: "Automatic maintenance that doesn't respect GKE maintenance policies", "Maintenance windows", "Maintenance exclusions", "Node pool maintenance exclusions", "Limitations on configuring maintenance exclusions", "Example: Company's legacy database not resilient to node pool upgrades")
- Configure maintenance windows and exclusions (Sections: "Configure a maintenance window", "Configure a maintenance exclusion")
- Environment architecture (Sections: "Environment's cluster")
- Get support (Sections: "Before you contact support", "Recommendations for efficient support")
- Disruptions (Sections: "Voluntary and involuntary disruptions", "Dealing with disruptions", "Pod disruption budgets")
- Specifying a Disruption Budget for your Application (Sections: "Identify an Application to Protect", "Think about how your application reacts to disruptions", "Rounding logic when specifying percentages", "Specifying a PodDisruptionBudget", "Create the PDB object", "Check the status of the PDB", "Healthiness of a Pod", "Unhealthy Pod Eviction Policy", "Arbitrary workloads and arbitrary selectors")
- Configure Liveness, Readiness and Startup Probes (Sections: "Define readiness probes")
This skill creates no billable resources, so no teardown is required.
Diagnostic workflow
Scope and Managed Airflow check:
- Managed Service for Apache Airflow (formerly Cloud Composer): Under Environment's cluster, clusters for Managed Airflow and workloads deployed into them are managed by Google: do not introduce changes in the environment's cluster (doing so might break the environment). When the user says the cluster belongs to a Managed Airflow (or Cloud Composer) environment, do not propose workload or PDB changes; ask when that isn't clear.
- Unpermissive PDBs (
PDB_UNPERMISSIVE): For requests solely aboutPDB_UNPERMISSIVEor drains blocked by"Cannot evict pod as it would violate the pod's disruption budget."on a matching PDB, stop and point the user to When GKE identifies workloads with vulnerability to disruption and Troubleshoot node pool upgrades taking longer than usual to adjustminAvailableormaxUnavailablethemselves. In an audit, handle matching unpermissive PDBs via "When a matching PDB already exists".- Other GKE topics: Hand off general probe/PDB setup to
gke-reliability, upgrade planning togke-upgrades,PendingPods togke-workload-troubleshooting, autoscaling/stockouts togke-cluster-autoscaler, ComputeClasses togke-compute-classes, and general manifest generation togke-manifest-generation.- Provided manifests, insights, or
kubectloutput: After the Managed Airflow check, if insight/recommendation details, manifests, orkubectloutput are provided, capture any{project_id},{cluster_name}, and{location}, skip Steps 1–2 (honoring dismissed recommendations per Step 2), and evaluate every workload in Step 3 without discovery commands.
Step 1: Identify the target cluster
Obtain {project_id}, {cluster_name}, and {location} (control plane region
or zone) from the prompt. If {project_id} is unknown, ask the user; if
{cluster_name} or {location} is unknown, ask the user or follow
View your clusters
for {project_id}. Ask when it isn't clear whether {cluster_name} belongs to
a Managed Airflow environment.
Step 2: Check for PDB_UNPROTECTED_STATEFULSET insights and recommendations
- Query insights: Under
Identify workloads with vulnerability to disruption,
disruption-readiness insights are available only via
gcloudor the Recommender API (not the Google Cloud console). Followgcloudor API instructions in View insights and recommendations for{project_id}and{location}, filtering forPDB_UNPROTECTED_STATEFULSET. - Filter active insights and check recommendation state: Keep
ACTIVEinsights matching{cluster_name}via "State Info" and "Target Resources", record "Severity" and "Last Refresh Time", and read{statefulset_name}({namespace}if shown), suggested selector labels ("Description" and "Content"), andassociatedRecommendations.recommendationvia Insights, "View insights and recommendations" and "Dismiss recommendations using the Google Cloud CLI" in Optimize your usage of GKE with insights and recommendations, and "When GKE identifies workloads with vulnerability to disruption" and "Implement the guidance to improve disruption readiness" in Ensure workloads are disruption-ready. Check the recommendation's state using only the read-only list (ACTIVEfilter hides dismissed recommendations) or describe step in "Dismiss recommendations using the Google Cloud CLI"—never runmark-dismissed. If{namespace}or{statefulset_name}is unknown, ask or point to Inspecting a StatefulSet. - Branch on results (include the
.md.txtlinks from items 1–2 when pointing the user to those sections):- Active insights: For each active insight, if its associated
recommendation was dismissed, report that the user dismissed it instead
of recommending a PDB again (dismissing only hides it in the console
while it remains discoverable via
gcloudand the Recommender API; see Dismiss or restore recommendations). Otherwise continue to Step 3 for{statefulset_name}in{namespace}. - No active insight for
{cluster_name}: Recommendations are assessed once daily ("Implement the guidance to improve disruption readiness" in item 2), so recent changes may not appear yet. Proceed to Step 3 if{statefulset_name}and{namespace}were given; otherwise report none was found and ask for them (or point to "Inspecting a StatefulSet" in item 2). - API disabled or unreachable: If
recommender.googleapis.comis disabled, point to "gcloud" in Enable the API (and continue to Step 3 if{statefulset_name}and{namespace}are known). If unreachable, point to "View insights and recommendations" (item 1), "Inspecting a StatefulSet" (item 2), and Check the status of the PDB, continuing with Step 3 once output is shared.
- Active insights: For each active insight, if its associated
recommendation was dismissed, report that the user dismissed it instead
of recommending a PDB again (dismissing only hides it in the console
while it remains discoverable via
Step 3: Inspect the StatefulSet selector and existing PDBs
Configure kubectl credentials for {cluster_name} in {location} via
Store cluster information for kubectl,
and evaluate every StatefulSet in scope (never stop after the first):
- Inspect the StatefulSet: Follow "Inspecting a StatefulSet" (and see
"Creating a StatefulSet" for
spec.template.metadata.labels) in Deploying a stateful application for{statefulset_name}in{namespace}to read{replicas}(spec.replicas),{statefulset_selector}(.spec.selector), andspec.template.metadata.labels, plus containerreadinessProbe(see Define readiness probes). Per Identify an Application to Protect,.spec.selectorgoes into a matching PDB's.spec.selector. If{statefulset_name}is not found, ask whether it was deleted:- If the user confirms it was deleted, tell them no PDB is needed and the recommendation can take up to 24 hours to resolve (see Implement the guidance to improve disruption readiness).
- Otherwise, re-check
{namespace}and any Step 2 insight's Target Resources (it might target another cluster); if still not found, direct the user to Cloud Customer Care ("Before you contact support" and "Recommendations for efficient support" in Get support).
- Inspect PDBs and selector overlap in
{namespace}: Follow "Check the status of the PDB", "Specifying a PodDisruptionBudget", "Identify an Application to Protect", and "Arbitrary workloads and arbitrary selectors" in Specifying a Disruption Budget for your Application to read each PDB's.spec.selector({}matches every Pod in{namespace}),minAvailable,maxUnavailable,status.expectedPods, andstatus.disruptionsAllowed.PDB_UNPROTECTED_STATEFULSETalerts when no existing PDB labels match{statefulset_selector}(see When GKE identifies workloads with vulnerability to disruption). Count only a PDB whose.spec.selectorequals{statefulset_selector}as a match; record any matching{existing_pdb_name}, and record any{existing_pdb_name}selecting the Pods without matching ({}, subset labels, extraspec.template.metadata.labels, or a Pod subset), noting whether it also selects other workloads' Pods.
Step 4: Classify each evaluated StatefulSet from live state
Classify each workload into one category from Step 3, reporting any Step 2 Severity:
StatefulSet Protected:{existing_pdb_name}in{namespace}equals{statefulset_selector}(any{replicas}). Record{other_statefulset_name}and{other_statefulset_summary}via "When a matching PDB already exists".Single-Replica or Scaled-to-Zero StatefulSet:{replicas} <= 1and no PDB in{namespace}equals{statefulset_selector}. Record{other_statefulset_name}and{other_statefulset_summary}via "Single-replica and scaled-to-zero StatefulSets".Unprotected StatefulSet:{replicas} > 1and no PDB in{namespace}equals{statefulset_selector}. Record missing-PDB or{existing_pdb_name}mismatch details in{classification_summary}; follow "Unprotected StatefulSet (add or update a PDB)".
Count the Unprotected StatefulSet workloads as {count}.
Remediation
Give the user fixes to apply themselves (never modify resources or state).
Unprotected StatefulSet (add or update a PDB)
- Specify
.spec.selectorand disruption budget: Point to "When GKE identifies workloads with vulnerability to disruption" and "Implement the guidance to improve disruption readiness" in Ensure workloads are disruption-ready, Pod disruption budgets, and "Identify an Application to Protect", "Think about how your application reacts to disruptions", "Rounding logic when specifying percentages", and "Specifying a PodDisruptionBudget" in Specifying a Disruption Budget for your Application:- Selector: Copy
{statefulset_selector}(.spec.selector, matching Step 2's recommendation when current) into the PDB's.spec.selector, without adding Pod-template-onlyspec.template.metadata.labels(see Creating a StatefulSet). - Budget: Set only one of
maxUnavailable(1adapts to scale; requires one shared controller) orminAvailable(quorum, such as2of3or3of5). Never setmaxUnavailableto0/0%orminAvailableto100%or an integer/rounded-up percentage not less than{replicas}(PDB_UNPERMISSIVE, causing 1-hour drain waits).
- Selector: Copy
- Create vs. update overlapping
{existing_pdb_name}: Under "Identify an Application to Protect", "Specifying a PodDisruptionBudget", "Create the PDB object", and "Arbitrary workloads and arbitrary selectors" in Specifying a Disruption Budget for your Application (eviction is disallowed for any Pod covered by multiple PDBs):- No PDB selects the Pods: Point to "Create the PDB object" to add it.
{existing_pdb_name}selects only this StatefulSet's Pods (empty{}, subset labels, extraspec.template.metadata.labels, or a Pod subset): Advise not creating a second PDB; point to "Create the PDB object" to update{existing_pdb_name}in place to{statefulset_selector}.{existing_pdb_name}also selects other workloads' Pods (empty{}, shared subset labels, or a shared Pod-template label): Warn thatmaxUnavailablerequires all selected Pods to share one controller (bare or unsupported-controller Pods allow only integerminAvailable), the budget is shared across all selected Pods rather than per-workload quorum, and adding a second PDB blocks eviction while narrowing{existing_pdb_name}alone drops coverage from the other workloads; point to those sections to give each controller a non-overlapping PDB matching.spec.selector.
- Readiness probes (
PDB_STATEFULSET_WITHOUT_PROBES): PDBs respect readiness probes when they look at which Pods can be counted as healthy, and a Pod counts as healthy when it has a.status.conditionsitem withtype="Ready"andstatus="True". If missing, point to Healthiness of a Pod, When GKE identifies workloads with vulnerability to disruption, Define readiness probes, and Updating a StatefulSet. - Resolution window and dismissal: Recommendations are assessed once daily and take up to 24 hours to resolve after changes (see Implement the guidance to improve disruption readiness). Only if the user decides not to add a PDB, point to "Dismiss or restore recommendations" and "Dismiss recommendations using the Google Cloud CLI" in Optimize your usage of GKE with insights and recommendations: dismissing only hides a recommendation for all users in the console (where it doesn't appear again even if GKE detects the same behavior), while it remains discoverable via the Google Cloud CLI and Recommender API.
Single-replica and scaled-to-zero StatefulSets
- If
{replicas} == 0: Ask whether it stays at0or scales up (under Check the status of the PDB, zero matching Pods showsALLOWED DISRUPTIONS: 0). If kept at0, no Pods run to disrupt (for dismissal, see item 4 of "Unprotected StatefulSet (add or update a PDB)" above); if scaling to1, follow items 2–3; if scaling above1(see Scaling a StatefulSet), follow "Unprotected StatefulSet (add or update a PDB)" for that target. - Explain 1-replica PDB trade-offs: PDBs target replicated apps ("Dealing
with disruptions" and "Pod disruption budgets" in
Disruptions).
Under "Think about how your application reacts to disruptions" and
"Specifying a PodDisruptionBudget" (
Example 4) in Specifying a Disruption Budget for your Application,maxUnavailable: 1or a positive percentage still allows evicting the lone Pod (100%unavailability), whilemaxUnavailableof0/0%orminAvailableof1/100%triggersPDB_UNPERMISSIVEand a 1-hour drain wait on Standard surge and Autopilot upgrades (see item 3 of "When a matching PDB already exists" below). If a non-matching{existing_pdb_name}selects the Pod, handle overlap per item 2 of "Unprotected StatefulSet (add or update a PDB)". - Documented single-instance options: Point the user to:
- Omit a PDB and tolerate brief drain downtime (see Think about how your application reacts to disruptions). If an active recommendation exists and the user decides against a PDB, share item 4 of "Unprotected StatefulSet (add or update a PDB)" above.
- Scale above
1replica before adding a PDB via Scaling a StatefulSet. - Control maintenance timing via a cluster maintenance window/exclusion or (Standard only) a node-pool maintenance exclusion—noting maintenance policies don't block all automatic maintenance and on Autopilot GKE honors the configuration if possible; see "Automatic maintenance that doesn't respect GKE maintenance policies", "Maintenance windows", "Maintenance exclusions", "Node pool maintenance exclusions", "Limitations on configuring maintenance exclusions", and "Example: Company's legacy database not resilient to node pool upgrades" in Maintenance windows and exclusions, Automatic node upgrades, and "Configure a maintenance window" and "Configure a maintenance exclusion" in Configure maintenance windows and exclusions.
When a matching PDB already exists
- Stale active insight: If an active
PDB_UNPROTECTED_STATEFULSETinsight still appears for{other_statefulset_name}, ask when{existing_pdb_name}was created or updated and check "Target Resources" and "Last Refresh Time" in Insights. If "Last Refresh Time" is earlier than the PDB change, explain the daily 24-hour assessment window (see Implement the guidance to improve disruption readiness) and advise not dismissing just to clear normal lag: under Dismiss or restore recommendations, dismissing only hides a recommendation for all users in the console (where it doesn't appear again even if GKE detects the same behavior), while it remains discoverable viagcloudand the Recommender API. If "Last Refresh Time" is later than the PDB change and the insight is still active, direct the user to Cloud Customer Care via "Before you contact support" and "Recommendations for efficient support" in Get support. - Single or zero replicas: If
{replicas} <= 1, on1replica even a permissive PDB allows100%unavailability (0matches zero Pods); follow "Single-replica and scaled-to-zero StatefulSets".
Check—or point the user to check:
- Permissiveness (
PDB_UNPERMISSIVE): Verify via "Check the status of the PDB", "Rounding logic when specifying percentages", and "Specifying a PodDisruptionBudget" in Specifying a Disruption Budget for your Application that when{replicas} > 0and Pods are healthy,ALLOWED DISRUPTIONS(status.disruptionsAllowed) is non-zero (minAvailable, rounded up for percentages, is less than{replicas}, ormaxUnavailable > 0); otherwise point to When GKE identifies workloads with vulnerability to disruption and Troubleshoot node pool upgrades taking longer than usual. - Readiness probes (
PDB_STATEFULSET_WITHOUT_PROBES): Check containerreadinessProbe; if missing, point to item 3 of "Unprotected StatefulSet (add or update a PDB)" above. - Disruption despite both: Point the user to:
- One-hour surge PDB timeout ("maxSurge: GKE creates a new surge node before removing an existing one" and "maxUnavailable: GKE makes an existing node unavailable to recreate it") and Standard blue-green pool deletion bypassing PDBs ("Phase 3: Drain blue pool" and "Phase 5: Delete blue pool") in Node upgrade strategies, plus the 1-hour PDB limit in Automatic node upgrades and When GKE identifies workloads with vulnerability to disruption.
unhealthyPodEvictionPolicy: AlwaysAllowevicting running non-Ready Pods regardless of budget (see Unhealthy Pod Eviction Policy).- Involuntary disruptions or direct deletions bypassing PDBs ("Voluntary and involuntary disruptions" and "Pod disruption budgets" in Disruptions).
- Escalation: Direct the user to Cloud Customer Care via "Before you contact support" and "Recommendations for efficient support" in Get support.
Output format
Cite relied-on doc sections by exact title and link. Omit unprovided
{cluster_name}, {location}, or {project_id}; omit Unprotected StatefulSets detected and Recommended remediation when {count} is 0; omit
Other evaluated StatefulSets when none apply ({other_statefulset_summary}
covers single-replica/scaled-to-zero options or {existing_pdb_name}
permissiveness, readiness-probe, and 24-hour lag notes); and report Step 2's
Severity
when retrieved.
### Unprotected StatefulSets detected
Found {count} `StatefulSet` workload(s) without a matching
`PodDisruptionBudget` on cluster `{cluster_name}` (`{location}`, Project:
`{project_id}`):
| StatefulSet | Namespace | Replicas | Classification & details |
| :--- | :--- | :---: | :--- |
| `{statefulset_name}` | `{namespace}` | {replicas} | {classification_summary} |
### Other evaluated StatefulSets
* `{other_statefulset_name}` (`{namespace}`, `replicas: {replicas}`):
{other_statefulset_summary}
### Recommended remediation
* For `{statefulset_name}` (`{namespace}`): follow "Identify an Application
to Protect", "Specifying a PodDisruptionBudget", "Create the PDB object",
and "Arbitrary workloads and arbitrary selectors" in
[Specifying a Disruption Budget for your Application](https://kubernetes.io/docs/tasks/run-application/configure-pdb/)
to create a `PodDisruptionBudget` copying `{statefulset_selector}` with an
appropriate budget, or update `{existing_pdb_name}` without overlapping
PDBs.
### Verification
* Follow
[Check the status of the PDB](https://kubernetes.io/docs/tasks/run-application/configure-pdb/)
to verify `ALLOWED DISRUPTIONS` (`status.disruptionsAllowed`) is non-zero
and `status.expectedPods` matches `{replicas}`.
* Recommendations are assessed once daily and take up to 24 hours to resolve
(see
[Implement the guidance to improve disruption readiness](https://docs.cloud.google.com/kubernetes-engine/docs/how-to/workload-disruption-readiness.md.txt)).