compute-resource-availability-troubleshooting
Diagnoses, mitigates, and prevents Google Compute Engine resource availability and capacity stockout errors (ZONE_RESOURCE_POOL_EXHAUSTED, ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS, and machine type or accelerator unavailable in zone). Use ONLY when VM creation, Managed Instance Group (MIG) scale-out, or stopped instance startup fails due to regional or zonal hardware capacity exhaustion for GPUs, TPUs, or specific CPU shapes. DO NOT read or use for IAM permission errors, project quota limit increases without stockouts, general greenfield VM creation, or unexpected host reboots on already-runn
Try your first task
A source-grounded starter task has not been verified for this skill yet.
SKILL.md
name: compute-resource-availability-troubleshootingdescription: >- Diagnoses, mitigates, and prevents Google Compute Engine resource availability and capacity stockout errors (ZONE_RESOURCE_POOL_EXHAUSTED, ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS, and machine type or accelerator unavailable in zone). Use ONLY when VM creation, Managed Instance Group (MIG) scale-out, or stopped instance startup fails due to regional or zonal hardware capacity exhaustion for GPUs, TPUs, or specific CPU shapes. DO NOT read or use for IAM permission errors, project quota limit increases without stockouts, general greenfield VM creation, or unexpected host reboots on already-running VMs.metadata: version: "1.0.0" category: ComputeCompute Engine Resource Availability and Stockout Troubleshooting
MCP Tools:
list_machine_types,list_accelerator_types,get_zone_operation,list_instance_group_managers,list_managed_instances,list_reservations,get_reservation_details
This skill diagnoses and resolves Google Compute Engine resource availability errors (capacity stockouts). It provides immediate tactical workarounds to get blocked workloads running and long-term architectural mitigations using Regional Managed Instance Group (MIG) distribution shapes, Instance Flexibility, Dynamic Workload Scheduler (DWS) Flex-Start, and On-Demand Capacity Reservations.
Usage Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read and follow the instructions in
Usage Attribution.
Prerequisites & Environment Setup
Any host, project, IAM, or Compute Engine resource modifications (installing
gcloud, configuring MCP, enabling APIs or Log Analytics, creating instances
or reservations, updating MIGs, or deleting resources) must be explicitly
approved by the user before execution.
- Google Cloud SDK: Verify
gcloudis installed (gcloud --version). If missing or outdated, ask the user for approval before installing via Install the Google Cloud SDK. - Compute Engine MCP Server & Capability Boundary: When the Compute Engine
MCP server is configured (see
Use the Compute Engine MCP server;
requires
roles/mcp.toolUser), prefer its read-only discovery tools before falling back togcloud:list_machine_typesandlist_accelerator_typesto check whether a machine shape or GPU accelerator is offered in a target zone.get_zone_operationto inspect a failed zonal operation'serror.errorsarray (ZONE_RESOURCE_POOL_EXHAUSTED,ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS, orQUOTA_EXCEEDED).list_instance_group_managersandlist_managed_instancesto inspect zonal MIGs and per-instancecurrentAction/lastAttempt.errors(note: all Compute Engine MCP MIG tools—list_instance_group_managers,list_managed_instances, andget_instance_group_manager_basic_info—only accept azoneparameter and cannot inspect regional MIGs; usegcloudwith--region={region}or REST for regional MIGs).list_reservationsandget_reservation_detailsto inspect existing zonal reservation capacity (specificReservation.countvs.specificReservation.inUseCount).- MCP Limitations (Use
gcloudor REST Instead): MCPcreate_instancedoes not exposeprovisioning_model(FLEX_START/SPOT),max_run_duration,instance_termination_action, orreservation_affinity. MCP also lacks tools for Cloud Logging queries (gcloud logging read), Spot Capacity Advisor (gcloud beta compute advice capacity), Bulk Insert (gcloud compute instances bulk create), MIG distribution updates (gcloud compute instance-groups managed update), and reservation creation or deletion (gcloud compute reservations create/delete). Always usegcloudor the Compute Engine REST API for those operations.
- Authentication: Local CLI (
gcloud auth login), Application Default Credentials (gcloud auth application-default login), or headless/CI-CD viaGOOGLE_APPLICATION_CREDENTIALSor Workload Identity Federation. - Project, Region, Billing & API Enablement:
Note: An active billing account attachment is required (even for Free Tier eligible resources such asgcloud config set project {project_id} gcloud config set compute/region {region} gcloud config set compute/zone {zone} gcloud services enable compute.googleapis.com logging.googleapis.come2-micro). Never hardcode regions or zones; confirm{project_id},{region}, and{zone}(defaulting to a well-supported region such asus-central1/us-central1-aif unspecified), and consult Google Cloud Locations to verify regional and zonal availability (preview features and specialized GPUs/TPUs are restricted to specific locations). - Required IAM Roles:
- Read-Only Diagnostics & Capacity Advice:
roles/compute.viewerandroles/logging.viewer. - Log Analytics SQL (
_Default._AllLogs):roles/logging.viewAccessor(plusroles/bigquery.jobUserandroles/bigquery.dataViewerwhen querying via a linked BigQuery dataset). - Instance & MIG Remediation:
roles/compute.instanceAdmin.v1(plusroles/iam.serviceAccountUserwhen attaching a service account to VMs). - Reservation Creation & Teardown:
roles/compute.admin(or a custom role grantingcompute.reservations.create,compute.reservations.update, andcompute.reservations.delete, which are not included inroles/compute.instanceAdmin.v1). - MCP Tool Invocation:
roles/mcp.toolUser.
- Read-Only Diagnostics & Capacity Advice:
Step 1: Verify Stockout Scope & Distinguish from Quota Errors
A. Core Domain Rules (Explain to User)
Before taking action, clarify these three foundational Compute Engine behaviors (see Troubleshooting resource availability errors):
- Stockout != Quota:
ZONE_RESOURCE_POOL_EXHAUSTED(andZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS) means physical hardware capacity for that specific resource pool in that specific zone is temporarily full. Increasing project quota (IAM & Admin > Quotas & System Limits) will not resolve a physical stockout. Conversely,QUOTA_EXCEEDEDmeans the project's administrative quota ceiling has been reached regardless of physical hardware availability. When returned,ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILSalso includeserrorInfo.reason(such asconfiguration_availabilityfor unsupported hardware combinations that will not succeed on retry) anderrorInfo.metadata.zonesAvailablelisting sibling zones with capacity. - Scoped to Specific Shape & Zone: A stockout applies only to the exact
machine shape or accelerator (for example,
g2-standard-4orc4-standard-32) in that zone (for example,us-central1-a). Sibling zones (us-central1-b,us-central1-c,us-central1-f) or smaller shapes (c4-standard-16) frequently have ample capacity. - Existing Running VMs Are Safe: Resource availability errors affect only
new capacity allocations (creating new VMs, starting stopped
TERMINATEDVMs, changing machine types, or MIG scale-out). Currently runningRUNNINGVMs in the zone are unaffected.
B. Query Cloud Audit Logs for Resource Availability vs. Quota Failures
Run gcloud logging read with an explicit --freshness window (gcloud logging read defaults to 1d, so multi-day investigations require
--freshness=7d or --freshness=14d). Inspect
protoPayload.response.error.errors[0].code (not .reason) to distinguish
ZONE_RESOURCE_POOL_EXHAUSTED / ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS
from QUOTA_EXCEEDED:
gcloud logging read \
'resource.type="gce_instance" AND log_id("cloudaudit.googleapis.com/activity") AND (protoPayload.status.message:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "does not have enough resources available" OR "resource pool exhausted" OR "does not exist in zone" OR "QUOTA_EXCEEDED" OR "Quota") OR protoPayload.response.error.errors.code:("ZONE_RESOURCE_POOL_EXHAUSTED" OR "ZONE_RESOURCE_POOL_EXHAUSTED_WITH_DETAILS" OR "QUOTA_EXCEEDED"))' \
--project={project_id} --freshness=7d --limit=20 \
--format="table(timestamp, resource.labels.zone, protoPayload.resourceName, protoPayload.response.error.errors[0].code, protoPayload.status.message)"
- If
codeisQUOTA_EXCEEDED(Always Include Regional Quota Inspection): Do not treat this as a physical stockout. Always providegcloud compute regions describeto inspect regional quota limits and usage, and direct the user to IAM & Admin > Quotas & System Limits (or switch to a machine family with quota headroom):gcloud compute regions describe {region} --project={project_id} \ --format="table(quotas:format='table(metric,limit,usage)')" - If a MIG is failing to scale out: Also inspect MIG provisioning errors
directly (via Compute Engine MCP
list_managed_instancesfor zonal MIGs, orgcloud):# Zonal MIG (--zone={zone}) vs. Regional MIG (--region={region}): gcloud compute instance-groups managed list-errors {mig_name} \ --project={project_id} --zone={zone} gcloud compute instance-groups managed list-errors {mig_name} \ --project={project_id} --region={region}
C. Analyze Historical Stockout Frequency with Log Analytics SQL
Because failed cloudaudit.googleapis.com/activity entries have severity
ERROR (severity_number >= 500), Cloud Logging routes them to the
_Default log bucket ({project_id}.global._Default._AllLogs), whereas
_Required._AllLogs only stores Admin Activity logs below ERROR and System
Event logs.
- Prerequisite: Upgrade
_Defaultto Log Analytics (see Query and view logs in Log Analytics):gcloud logging buckets update _Default \ --project={project_id} --location=global --enable-analytics - JSON Column Extraction Rule: In
_AllLogs,resource.labelsandproto_payload.audit_log.requesthave BigQuery typeJSON. Never referenceresource.labels.zonedirectly inGROUP BY—always extract scalarSTRINGvalues withJSON_VALUE(...):SELECT JSON_VALUE(resource.labels.zone) AS zone, REGEXP_EXTRACT(JSON_VALUE(proto_payload.audit_log.request.machineType), r'([^/]+)$') AS machine_type, proto_payload.audit_log.status.message AS status_message, COUNT(*) AS stockout_count, MIN(timestamp) AS first_seen, MAX(timestamp) AS last_seen FROM `{project_id}.global._Default._AllLogs` WHERE timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 14 DAY) AND log_id = 'cloudaudit.googleapis.com/activity' AND resource.type = 'gce_instance' AND ( proto_payload.audit_log.status.message LIKE '%ZONE_RESOURCE_POOL_EXHAUSTED%' OR proto_payload.audit_log.status.message LIKE '%does not have enough resources available%' OR TO_JSON_STRING(proto_payload.audit_log.response.error.errors) LIKE '%ZONE_RESOURCE_POOL_EXHAUSTED%' ) GROUP BY zone, machine_type, status_message ORDER BY stockout_count DESC;
Step 2: Immediate Tactical Mitigations (Unblock Workload Now)
Execute these five steps in priority order to provision the blocked workload immediately:
1. Discover Sibling Zones Supporting the Machine Shape or GPU
If a zone returns ZONE_RESOURCE_POOL_EXHAUSTED or Machine type with name '...' does not exist in zone '...', check which sibling zones in {region}
offer the target machine shape and (if applicable) GPU accelerator using
Compute Engine MCP list_machine_types / list_accelerator_types or gcloud:
# List all zones in {region} that offer {machine_type}:
gcloud compute machine-types list --project={project_id} \
--filter="name={machine_type} AND zone ~ ^{region}-" \
--format="table(name, zone, guestCpus, memoryMb)"
# For GPU workloads (especially N1 with attached GPUs), verify accelerator zones:
gcloud compute accelerator-types list --project={project_id} \
--filter="name={accelerator_type} AND zone ~ ^{region}-" \
--format="table(name, zone, maximumCardsPerInstance)"
2. Use Bulk Insert (bulk create) to Pick an Available Zone
Instead of guessing and retrying individual zones manually, use gcloud compute instances bulk create with --region={region} and
--target-distribution-shape=ANY_SINGLE_ZONE (for colocated VMs or --count=1)
or --target-distribution-shape=BALANCED (to spread across available zones).
Compute Engine automatically selects zone(s) in {region} with capacity (see
Create VMs in bulk):
# On instances bulk create, --target-distribution-shape is UPPERCASE (ANY, ANY_SINGLE_ZONE, BALANCED):
gcloud compute instances bulk create \
--project={project_id} --region={region} \
--name-pattern="{vm_name}-#" --count={vm_count} \
--target-distribution-shape=ANY_SINGLE_ZONE \
--machine-type={machine_type}
3. Check Spot VM Obtainability via Spot Capacity Advisor (advice capacity)
When provisioning Spot VMs, use gcloud beta compute advice capacity (the
Spot Capacity Advisor API) to compare real-time Spot obtainability scores and
estimated uptime (estimatedUptime) across zones and machine types in
{region} before creating instances (use
gcloud beta compute advice capacity-history for 30-day historical preemption
rates; see
View the availability of Spot VMs):
- Critical Scope & Flag Rules:
gcloud beta compute advice capacitysupports only Spot VMs (--provisioning-model=SPOT, the only supported value; it cannot querySTANDARDon-demand capacity).- There is no
--machine-typeflag onadvice capacity—specify machine types with--instance-selection-machine-types={machine_type_1},{machine_type_2}or--instance-selection="name={selection_name},machine-type={machine_type}". - On
advice capacity,--target-distribution-shapechoices are UPPERCASE (ANY,ANY_SINGLE_ZONE, orBALANCED). Always include--provisioning-model=SPOT,--instance-selection-machine-types, and uppercase--target-distribution-shape=ANY(orANY_SINGLE_ZONE/BALANCED):
gcloud beta compute advice capacity \
--project={project_id} --region={region} \
--provisioning-model=SPOT --size={vm_count} \
--instance-selection-machine-types={machine_type_1},{machine_type_2} \
--target-distribution-shape=ANY
4. Right-Size Large Shapes (> 32 vCPUs)
Physical host fragmentation makes large machine shapes (such as *-standard-64,
*-standard-96, or *-standard-128) significantly scarcer than smaller shapes.
If the workload can scale horizontally, split a single 64-vCPU request into two
32-vCPU VMs or four 16-vCPU VMs.
5. Use Dynamic Workload Scheduler (DWS) Flex-Start for Scarce GPUs
When scarce GPU shapes (a4-highgpu-8g, a3-highgpu-8g, a2-highgpu-1g, or
g2-standard-4) fail immediate creation with ZONE_RESOURCE_POOL_EXHAUSTED,
use Dynamic Workload Scheduler (DWS) Flex-Start to queue and automatically
provision capacity when hardware frees up (see
About Dynamic Workload Scheduler
and
Create a standalone Flex-start VM):
- Mandatory CLI Rules for DWS Flex-Start Standalone VMs:
- Machine Family Accelerator & Maintenance Rule: For
G2(nvidia-l4),G4,A2(nvidia-tesla-a100),A3, andA4machine series, GPUs are automatically attached by the machine type (omit--acceleratorunless attaching GPUs toN1or using vWS). Standalone Flex-Start GPU creation requires--reservation-affinity=none,--max-run-duration(10mto7d),--instance-termination-action=DELETE(orSTOP),--maintenance-policy=TERMINATE, and--request-valid-for-duration(90sto2hto queue in a zone, or0sfor immediate-only allocation). - For multi-instance GPU jobs or longer/multi-zone queueing, use a MIG
with a Flex-Start instance template and resize requests (see
Create resize requests in a MIG).
The MIG requires autoscaling disabled,
--default-action-on-vm-failure=do-nothing, and (on a Regional MIG)--target-distribution-shape=any-single-zonewith--instance-redistribution-type=none.
- Machine Family Accelerator & Maintenance Rule: For
# Option A: Standalone VM with DWS Flex-Start (queues up to 2h via --request-valid-for-duration):
gcloud compute instances create {vm_name} \
--project={project_id} --zone={zone} \
--machine-type=g2-standard-4 \
--provisioning-model=FLEX_START \
--reservation-affinity=none \
--request-valid-for-duration=2h \
--max-run-duration=7d \
--instance-termination-action=DELETE \
--maintenance-policy=TERMINATE
# Option B: Instance Template + Zonal or Regional MIG Resize Request (longer queueing):
gcloud compute instance-templates create {template_name} \
--project={project_id} --machine-type=g2-standard-4 \
--provisioning-model=FLEX_START \
--reservation-affinity=none \
--max-run-duration=7d \
--instance-termination-action=DELETE \
--maintenance-policy=TERMINATE
# Queue capacity on a MIG configured per bullet 2 (--zone={zone} or --region={region}):
gcloud compute instance-groups managed resize-requests create {mig_name} \
--project={project_id} --region={region} \
--resize-request={request_name} --resize-by={vm_count}
Step 3: Long-Term Architectural Prevention
To permanently eliminate single-zone stockouts for production services, implement one or more of the following patterns:
A. Configure Regional MIGs with balanced or any Distribution Shape
A Regional MIG spreads instances across zones in a region. Configure its Target Distribution Shape (see Regional MIG distribution shape documentation):
balanced(BALANCED): Prioritizes even zonal distribution, but automatically spills over to healthy sibling zones during a stockout.any(ANY): Provisions capacity in whichever zone has available hardware fastest, maximizing obtainability under regional pressure.any-single-zone(ANY_SINGLE_ZONE): Colocates all VMs in one healthy zone with sufficient capacity (ideal for low-latency inter-node workloads).
Critical Operational Rule (--instance-redistribution-type):
- Regional MIGs default to
even(EVEN) +proactive(PROACTIVE), which periodically deletes running VMs in over-represented zones and recreates them in under-represented or recovering zones to enforce strict zonal balance—disrupting stateful workloads and failing during stockouts. balanced(BALANCED),any(ANY), andany-single-zone(ANY_SINGLE_ZONE) all require--instance-redistribution-type=none. Compute Engine supportsproactiveredistribution only withevenand rejectsproactivewithbalanced,any, orany-single-zone(see Set a target distribution shape).- Calliope Flag Case Rule: Unlike
instances bulk create(uppercase),gcloud compute instance-groups managed updaterequires lowercase values for--target-distribution-shape(any,any-single-zone,balanced,even) and--instance-redistribution-type(none,proactive):
# Allow spillover to healthy zones without proactively deleting running VMs:
gcloud compute instance-groups managed update {mig_name} \
--project={project_id} --region={region} \
--instance-redistribution-type=none --target-distribution-shape=balanced
# Or maximize obtainability in any zone (requires --instance-redistribution-type=none):
gcloud compute instance-groups managed update {mig_name} \
--project={project_id} --region={region} \
--instance-redistribution-type=none --target-distribution-shape=any
B. Enable MIG Instance Flexibility (Multi-Family Machine Type Fallback)
Configure the MIG with Instance Flexibility (instanceFlexibilityPolicy)
so it can fall back across equivalent machine families (for example, rank 1
c3-standard-8, rank 2 n2-standard-8 or n2d-standard-8) when the primary
family is exhausted (see
About instance flexibility
and
Add instance flexibility):
- Constraints & Disk/Repair Rules:
- Supported only on Regional MIGs (not Zonal MIGs) with
--target-distribution-shapeset tobalanced,any, orany-single-zone(defaultevenis rejected) and--instance-redistribution-type=none. Configure withgcloud beta compute instance-groups managed update(--instance-selection/--remove-instance-selections) or RESTinstanceFlexibilityPolicy.instanceSelections. - Falling back across series with incompatible boot disk support (such as
Hyperdisk-only
n4-standard-8to Persistent-Disk-onlyn2-standard-8) requires per-selection disk overrides via--instance-flexibility-policy(or--flags-file) rather than--instance-selectionalone. - Enable
--force-update-on-repairso MIG VM repairs can fall back to alternative machine types (and whenever using disk or CPU overrides).
- Supported only on Regional MIGs (not Zonal MIGs) with
gcloud beta compute instance-groups managed update {mig_name} \
--project={project_id} --region={region} \
--target-distribution-shape=balanced \
--instance-redistribution-type=none \
--instance-selection="name=primary-tier,machine-type=c3-standard-8,rank=1" \
--instance-selection="name=fallback-tier,machine-type=n2-standard-8,rank=2"
C. Purchase On-Demand Capacity Reservations
For mission-critical production workloads requiring 100% guaranteed zonal capacity assurance, create an On-Demand Zonal Reservation (see About reservations of zonal resources and Create a reservation for a single project):
- Operational & IAM Rules:
- Requires
roles/compute.admin(or customcompute.reservations.*) and succeeds only when physical capacity is currently available in{zone}—if{zone}is actively stocked out, create the reservation in an available sibling zone or wait until capacity replenishes. - For N1 with GPUs or Local SSDs,
--acceleratorand--local-ssdflags must exactly match the consuming VM's properties.
- Requires
gcloud compute reservations create {reservation_name} \
--project={project_id} --zone={zone} \
--vm-count={vm_count} --machine-type={machine_type}
Safe Rollback, Teardown & Cost Governance
- Restore Regional MIG Distribution Settings: To roll back from
anyto opportunistic zonal balance (balancedrequires--instance-redistribution-type=noneand converges on resizes), or to return a stateless MIG to stricteven+proactivedistribution (after manually rebalancing if zone counts differ by more than 1 VM):# Option A: Converge opportunistically on resizes without deleting VMs: gcloud compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --target-distribution-shape=balanced --instance-redistribution-type=none # Option B: Stateless strict even + proactive (rebalance manually first if >1 VM diff): gcloud compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --target-distribution-shape=even --instance-redistribution-type=none gcloud compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --instance-redistribution-type=proactive - Remove MIG Instance Selections: To roll back fallback selections:
gcloud beta compute instance-groups managed update {mig_name} \ --project={project_id} --region={region} \ --remove-instance-selections=fallback-tier - Delete Unused Capacity Reservations & Temporary VMs:
- Cost Warning: On-Demand Capacity Reservations are billed at standard Compute Engine rates from creation whether or not VMs run inside them (and count toward regional quota). Consult Google Cloud pricing and Compute Engine pricing for current rates rather than quoting hardcoded prices.
- Omit
--quieton manual destructive commands so interactive confirmation prevents accidental deletion in production:gcloud compute reservations delete {reservation_name} \ --project={project_id} --zone={zone} gcloud compute instances delete {vm_name} \ --project={project_id} --zone={zone}