compute-host-events-troubleshooting
Investigates, diagnoses, and mitigates Google Compute Engine virtual machine (VM) host events, including planned infrastructure maintenance (migrateOnHostMaintenance, terminateOnHostMaintenance), physical host hardware errors (hostError), guest OS terminations (guestTerminate), and Spot or Preemptible VM preemptions (compute.instances.preempted) using shutdown-script metadata. Use ONLY when an existing Compute Engine VM unexpectedly reboots, stops, crashes, is preempted, or becomes unresponsive due to physical host maintenance, hardware errors, or Local SSD recovery. DO NOT read or use for cre
Try your first task
A source-grounded starter task has not been verified for this skill yet.
SKILL.md
name: compute-host-events-troubleshootingdescription: >- Investigates, diagnoses, and mitigates Google Compute Engine virtual machine (VM) host events, including planned infrastructure maintenance (migrateOnHostMaintenance, terminateOnHostMaintenance), physical host hardware errors (hostError), guest OS terminations (guestTerminate), and Spot or Preemptible VM preemptions (compute.instances.preempted) using shutdown-script metadata. Use ONLY when an existing Compute Engine VM unexpectedly reboots, stops, crashes, is preempted, or becomes unresponsive due to physical host maintenance, hardware errors, or Local SSD recovery. DO NOT read or use for creating new standalone VMs, Managed Instance Group (MIG) autoscaler scale-out or scale-in failures, Google Kubernetes Engine (GKE) clusters, application OOM debugging, or VPC firewall issues.metadata: version: "1.0.0" category: ComputeCompute Engine VM Host Events & Infrastructure Crash Troubleshooting
This skill diagnoses unexpected Google Compute Engine VM restarts,
terminations, and unresponsiveness caused by infrastructure host events
(compute.instances.hostError, compute.instances.migrateOnHostMaintenance,
compute.instances.terminateOnHostMaintenance, compute.instances.preempted,
compute.instances.automaticRestart) or guest-OS-initiated shutdowns
(compute.instances.guestTerminate). It provides authoritative Cloud Audit Log
queries, Log Analytics _Required._AllLogs SQL aggregations, upcoming
maintenance window checks, and host scheduling remediations.
MCP Tools:
get_instance_basic_info,list_instances,list_instance_attached_disks,start_instance,stop_instance,reset_instance,delete_instance,get_zone_operation
Usage Attribution
MUST READ: Before executing any gcloud, REST API, or MCP command, you
MUST read and follow the instructions in
Usage Attribution.
Prerequisites & Environment Setup
Any host or project modifications (installing or updating gcloud, configuring
the Compute Engine MCP server, enabling APIs or Log Analytics, mutating VM
scheduling/metadata, creating log-based metrics, or deleting resources) must be
explicitly approved by the user before execution.
- Google Cloud SDK: Verify
gcloudis installed (gcloud --version). If missing or outdated, ask the user for approval before installing via Install the gcloud CLI. - Compute Engine MCP Server & Capability Boundary:
- The Compute Engine MCP server
(Use the Compute Engine MCP server)
exposes zonal VM discovery and lifecycle tools (
get_instance_basic_info,list_instances,list_instance_attached_disks,start_instance,stop_instance,reset_instance,delete_instance,get_zone_operation, requiringroles/mcp.toolUser). - Critical Routing Rule:
get_instance_basic_inforeturns a reducedInstanceBasicInfoprojection (name,id,status,machine_type,create_time,guest_accelerators) that omitsschedulingandresourceStatus.upcomingMaintenance, and Compute Engine MCP does not expose tools for Cloud Logging queries, serial console output,set-scheduling,perform-maintenance, or instance metadata updates. Always route full scheduling inspection, upcoming maintenance checks, serial port output, audit log queries, and scheduling/metadata mutations togcloud(or Compute Engine / Cloud Logging REST APIs).
- The Compute Engine MCP server
(Use the Compute Engine MCP server)
exposes zonal VM discovery and lifecycle tools (
- Authentication:
- Local CLI:
gcloud auth login - Application Default Credentials:
gcloud auth application-default login - Service Accounts: Set
GOOGLE_APPLICATION_CREDENTIALSto your service account credentials path.
- Local CLI:
- Project, Region & Billing:
Note: An active billing account attachment is required (even when evaluating Free Tier eligible resources). If no region is specified, default togcloud config set project {project_id} gcloud config set compute/region {region} gcloud config set compute/zone {zone}us-central1(us-central1-a), and check Google Cloud locations for regional hardware availability. - API Enablement:
gcloud services enable compute.googleapis.com logging.googleapis.com monitoring.googleapis.com - Required IAM Roles:
- Read-Only Diagnostics:
roles/compute.viewer(instance scheduling and serial console output),roles/logging.viewer(orroles/logging.privateLogViewerto readsystem_eventandactivityaudit logs), androles/logging.viewAccessorfor Log Analytics queries in Cloud Logging (plusroles/bigquery.jobUserandroles/bigquery.dataViewerwhen querying a linked Log Analytics dataset from BigQuery Studio orbq). - Remediation & MCP:
roles/compute.instanceAdmin.v1(to runset-scheduling,perform-maintenance,add-metadata, orremove-metadata),roles/logging.configWriter(to create log-based metrics), androles/mcp.toolUser(for Compute Engine MCP calls).
- Read-Only Diagnostics:
Diagnostic Workflow: Classify the Termination Event
When a user reports an unexpected VM reboot, stop, or crash, run targeted Cloud
Logging queries against cloudaudit.googleapis.com/system_event (Compute Engine
infrastructure events) and cloudaudit.googleapis.com/activity
(caller-initiated API stops/resets). Always pass --freshness=7d (or a window
covering the incident timestamp) because gcloud logging read defaults
--freshness to 1d and silently returns zero rows for incidents older than 24
hours. See
Troubleshooting VM shutdowns and reboots.
Key Distinction: hostError vs. Planned Maintenance vs. preempted
compute.instances.hostError(Unplanned Hardware/Hypervisor Crash): Indicates an unexpected hardware or hypervisor failure on the physical server hosting the VM. Ifscheduling.automaticRestartistrue(--restart-on-failure), Compute Engine automatically restarts the VM on a healthy physical host (typically within 3 to 5.5 minutes from detection on standard VMs, loggingcompute.instances.automaticRestart). On VMs with attached Local SSDs, Compute Engine logscompute.instances.hostErrorwithoperation.first="true"when Local SSD recovery begins andoperation.last="true"when recovery completes or reachesscheduling.localSsdRecoveryTimeout.compute.instances.migrateOnHostMaintenance/compute.instances.terminateOnHostMaintenance(Planned Infrastructure Maintenance): Compute Engine logs these exactsystem_eventmethod names (along withcompute.instances.upcomingMaintenancenotifications) during scheduled hardware or hypervisor updates. Depending ononHostMaintenance, the VM either transparently live-migrates (MIGRATE->compute.instances.migrateOnHostMaintenance) or terminates (TERMINATE->compute.instances.terminateOnHostMaintenance, followed bycompute.instances.automaticRestartifautomaticRestart: true).compute.instances.preempted(Spot / Preemptible VM Reclaimed): Indicates Compute Engine reclaimed capacity from a Spot VM or Preemptible VM.
1. Query Infrastructure Host Events & Caller Stops in Cloud Logging
Run this gcloud logging read command to determine whether {vm_name}
experienced a physical host error, planned maintenance, Spot preemption, MIG
recreation, or an external caller-initiated API stop/reset:
gcloud logging read 'resource.type="gce_instance" AND (log_id("cloudaudit.googleapis.com/system_event") OR log_id("cloudaudit.googleapis.com/activity")) AND protoPayload.methodName=~"(compute\.instances\.(hostError|migrateOnHostMaintenance|terminateOnHostMaintenance|preempted|automaticRestart|upcomingMaintenance|guestTerminate|repair.*)|v1\.compute\.instances\.(stop|reset|delete|performMaintenance|simulateMaintenanceEvent))" AND protoPayload.resourceName:"{vm_name}"' \
--project={project_id} \
--freshness=7d \
--limit=25 \
--format="table(timestamp, logName.basename(), protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, operation.first, operation.last, protoPayload.status.message)"
2. Check Specifically for Guest OS Initiated Terminations (guestTerminate)
If the VM stopped without a hostError, maintenance termination, preemption, or
user-initiated v1.compute.instances.stop Admin Activity log, check whether the
shutdown was initiated from inside the Guest OS (sudo shutdown -h now,
poweroff, systemctl poweroff, or an internal watchdog script):
gcloud logging read 'resource.type="gce_instance" AND log_id("cloudaudit.googleapis.com/system_event") AND protoPayload.methodName="compute.instances.guestTerminate" AND protoPayload.resourceName:"{vm_name}"' \
--project={project_id} \
--freshness=7d \
--limit=20 \
--format="yaml(timestamp, protoPayload.methodName, protoPayload.authenticationInfo.principalEmail, protoPayload.status)"
- Critical
guestTerminateAttribution Nuance: Becausecompute.instances.guestTerminateis asystem_eventgenerated when the guest ACPI power-off reaches the hypervisor, Compute Engine always records its generic built-in system principal inprotoPayload.authenticationInfo.principalEmail— it never identifies which Linux/Windows user, cron job, or script inside the VM ranshutdown/poweroff(whereas external caller-initiatedv1.compute.instances.stop,reset, ordeleteentries incloudaudit.googleapis.com/activityrecord the external IAM user or service account inprincipalEmail). - How to Find the In-Guest Culprit: Inspect the VM's serial console output
and guest OS logs from the previous boot (
journalctl -b -1,/var/log/syslogor/var/log/messages,/var/log/auth.log, andlast -x shutdown reboot):gcloud compute instances get-serial-port-output {vm_name} --project={project_id} --zone={zone} --port=1
3. Log Analytics SQL Query: Aggregate Host Events Across the Project
Compute Engine host events (hostError, migrateOnHostMaintenance,
terminateOnHostMaintenance, preempted, automaticRestart,
guestTerminate) are cloudaudit.googleapis.com/system_event logs. By default,
Cloud Logging's _Default sink excludes system_event logs and routes them
exclusively to the _Required log bucket.
Prerequisites for
_Required._AllLogsSQL:
- Upgrade the
_Requiredbucket to Log Analytics (seek user approval first):gcloud logging buckets update _Required --project={project_id} --location=global --enable-analytics- In Log Analytics
_AllLogsschemas,resource.labelshas BigQuery typeJSON(notSTRUCT). Always extract fields withJSON_VALUE(resource.labels.zone)andJSON_VALUE(resource.labels.instance_id)— grouping directly on rawJSONexpressions fails withGrouping by expressions of type JSON is not allowed.
SELECT
JSON_VALUE(resource.labels.zone) AS zone,
JSON_VALUE(resource.labels.instance_id) AS instance_id,
REGEXP_EXTRACT(proto_payload.audit_log.resource_name, r'[^/]+$') AS vm_name,
proto_payload.audit_log.method_name AS event_type,
COUNT(*) AS event_count
FROM `{project_id}.global._Required._AllLogs`
WHERE
resource.type = 'gce_instance'
AND timestamp >= TIMESTAMP_SUB(CURRENT_TIMESTAMP(), INTERVAL 30 DAY)
AND proto_payload.audit_log.method_name IN (
'compute.instances.hostError',
'compute.instances.migrateOnHostMaintenance',
'compute.instances.terminateOnHostMaintenance',
'compute.instances.preempted',
'compute.instances.automaticRestart',
'compute.instances.guestTerminate'
)
GROUP BY zone, instance_id, vm_name, event_type
ORDER BY event_count DESC;
4. Inspect Upcoming Scheduled Host Maintenance & Trigger Early Maintenance
For instances that emit advance maintenance notifications (such as A4X Max,
A4X, A4, A3, A2, G4, G2 GPU instances, bare metal -metal
instances, H4D / Cloud RDMA instances, X4 instances, storage-optimized Z3
/ Z4D / Z4M Local SSD instances, or Cloud TPU nodes), inspect upcoming
maintenance via gcloud or the in-guest metadata server. See
Monitor and plan for a host maintenance event
and
Get live migration notices from the metadata server:
# 1. Check a single instance for scheduling policy and upcoming maintenance (Compute Engine API):
gcloud compute instances describe {vm_name} --project={project_id} --zone={zone} \
--format="yaml(name, scheduling, resourceStatus.upcomingMaintenance)"
# 2. List upcoming maintenance windows across all instances in the project:
gcloud compute instances list --project={project_id} \
--format="table(name, zone.basename(), resourceStatus.upcomingMaintenance.type, resourceStatus.upcomingMaintenance.windowStartTime, resourceStatus.upcomingMaintenance.windowEndTime, resourceStatus.upcomingMaintenance.canReschedule, resourceStatus.upcomingMaintenance.maintenanceStatus)"
# 3. Query from inside the Guest OS via the Metadata Server:
curl -s "http://metadata.google.internal/computeMetadata/v1/instance/upcoming-maintenance?alt=json" \
-H "Metadata-Flavor: Google"
- API vs. Metadata Server JSON Field Name Difference:
- In
gcloud/ Compute Engine REST (resourceStatus.upcomingMaintenance) and Cloud Logging, the maintenance classification field is namedtype(SCHEDULEDorUNSCHEDULED), alongsidemaintenanceStatus(PENDINGorONGOING),canReschedule(trueorfalse),windowStartTime,windowEndTime, andlatestWindowStartTime. - When querying
/computeMetadata/v1/instance/upcoming-maintenance?alt=jsoninside the guest OS, returns"NONE"when no maintenance is scheduled, or a JSON object where the classification property is namedmaintenanceType(e.g.,"maintenanceType": "SCHEDULED").
- In
- Manually Trigger Host Maintenance Early (
perform-maintenance):- Preconditions: Check
canRescheduleandmaintenanceStatusfirst. You can runperform-maintenancewhencanRescheduleistrue(andmaintenanceStatusisPENDINGwhen an active notice exists, or on supported machine series before a notice is posted to reset the maintenance cycle). If an active notice reportscanReschedule: falseormaintenanceStatus: ONGOING, manual triggering is rejected by the API. - Immediate Impact: Confirm with the user before running —
perform-maintenancestarts host maintenance immediately (terminating and optionally restartingTERMINATE-policy instances right away). See Simulate or trigger a host maintenance event:
- Preconditions: Check
gcloud compute instances perform-maintenance {vm_name} --project={project_id} --zone={zone}
Actionable Remediation & Hardening Rules
Required IAM Role: Modifying instance scheduling or metadata requires
roles/compute.instanceAdmin.v1. Always obtain user confirmation before
executing mutating commands.
Case A: Preventing Downtime During Planned Maintenance (MIGRATE Policy)
If a live-migration-capable VM (such as an N2, N2D, C3, C3D, C4, or
N4 instance; note that standard E2 VMs only support MIGRATE during
host maintenance events and disallow TERMINATE) logged
compute.instances.terminateOnHostMaintenance during routine host maintenance,
its scheduling.onHostMaintenance policy was set to TERMINATE instead of
MIGRATE.
- Fix: Enable Live Migration (
--maintenance-policy=MIGRATE) and Automatic Restart (--restart-on-failure, which sets RESTscheduling.automaticRestart: true; use--no-restart-on-failureto disable). Per Set VM host maintenance policy,gcloud compute instances set-schedulingupdatesonHostMaintenance,automaticRestart(except on running CSEK-encrypted VMs),hostErrorTimeoutSeconds, andlocalSsdRecoveryTimeoutin-place on a running VM without stopping or rebooting it (note: although thegcloud compute instances set-scheduling --help/ RESTinstances.setSchedulingreference docstring contains a legacy note stating the method requires a stoppedTERMINATEDinstance, stopping the VM first is only required when changing--provisioning-model/--preemptibleor updatingautomaticRestarton CSEK-encrypted VMs):
gcloud compute instances set-scheduling {vm_name} \
--project={project_id} \
--zone={zone} \
--maintenance-policy=MIGRATE \
--restart-on-failure
- VM Families & Configurations That Do Not Support Live Migration (Always
Clarify):
Whenever recommending
--maintenance-policy=MIGRATE, explicitly note which VM configurations cannot live-migrate and instead require--maintenance-policy=TERMINATEpaired with--restart-on-failure(except Spot/Preemptible VMs, which requireTERMINATEwith--no-restart-on-failure), and note that standardE2VMs only supportMIGRATE:- Spot VMs (
provisioningModel: SPOT) and legacy Preemptible VMs - GPU-attached VMs (
A4X Max,A4X,A4,A3,A2,G4,G2, andN1with attached GPUs; note thatN2does not support attached GPUs) - Cloud TPU VMs and Bare Metal Instances (
*-metal) - Instances with a Cloud RDMA network interface (such as
H4DandZ4Mwith Cloud RDMA) X4andX5memory-optimized instances- Large Local SSD Configurations (
Z3instances with > 18,000 GiB of Local SSD, andZ4D/Z4Minstances with > 42,000 GiB of Local SSD)
- Spot VMs (
Case B: Faster Recovery from Unresponsive Hosts (--host-error-timeout-seconds)
If an application experiences prolonged unresponsiveness before Compute Engine's
default host watchdog declares a hostError, shorten the detection timeout via
--host-error-timeout-seconds:
- Constraints: Must be specified in 30-second increments. Valid range is
90to300seconds ingcloud(90to330seconds via Compute Engine RESTinstances.setScheduling). Unsupported on Sole-Tenant VMs and Spot / Preemptible VMs. Updates dynamically on a running VM (see Set VM host maintenance policy):
gcloud compute instances set-scheduling {vm_name} --project={project_id} --zone={zone} --host-error-timeout-seconds=90
Case C: Customizing Local SSD Recovery Timeout (--local-ssd-recovery-timeout)
When a hostError strikes a VM with attached Local SSDs, Compute Engine
attempts to recover data from the failed host's NVMe drives while the VM stays
offline, logging compute.instances.hostError with operation.first="true"
when recovery begins and operation.last="true" when recovery completes or
reaches scheduling.localSsdRecoveryTimeout.
- Valid Range, Defaults & Trade-off: Valid values are
0to168hours (up to 7 days) in 1-hour increments (see About host maintenance):- Defaults by Machine Series:
168hours (7 days):Z4Mbare metal andZ3bare metal (z3-highmem-192-highlssd-metal) instances6hours:Z4D,Z4M, andZ3VMs1hour: All other instance types
- Stateless Scratch / Cache Local SSDs: Set
--local-ssd-recovery-timeout=0to skip NVMe data recovery and restart the VM immediately with blank Local SSDs. - Stateful Local SSD Data: Increase
--local-ssd-recovery-timeout(up to168hours, e.g.,--local-ssd-recovery-timeout=24) to give Compute Engine more time to recover attached Local SSD data before restarting.
- Defaults by Machine Series:
# Fast restart (0 hours wait) for stateless cache/scratch Local SSD workloads:
gcloud compute instances set-scheduling {vm_name} --project={project_id} --zone={zone} --local-ssd-recovery-timeout=0
Case D: Handling Spot / Preemptible VM Preemptions (compute.instances.preempted)
When compute.instances.preempted appears in Cloud Logging, Compute Engine
reclaimed a Spot VM or Preemptible VM by sending an ACPI G2 Soft Off
signal.
- Best-Effort Preemption Window: The ACPI shutdown window provides up to 30 seconds on a best-effort basis (not guaranteed under severe physical capacity emergencies), so keep checkpointing scripts fast and idempotent and persist critical state continuously. See Spot VMs.
- Attach a
shutdown-script& Check the Metadata Server: Queryhttp://metadata.google.internal/computeMetadata/v1/instance/preemptedwith headerMetadata-Flavor: Google(TRUEduring preemption vs.FALSEon normal shutdown):
gcloud compute instances add-metadata {vm_name} \
--project={project_id} \
--zone={zone} \
--metadata=shutdown-script='#!/bin/bash
PREEMPTED=$(curl -s "http://metadata.google.internal/computeMetadata/v1/instance/preempted" -H "Metadata-Flavor: Google")
if [ "$PREEMPTED" = "TRUE" ]; then
echo "Spot VM preemption detected; flushing checkpoints..."
# Insert fast (<20s), idempotent checkpoint or drain command here
fi'
- Safe Rollback of Test Shutdown Metadata (Always Include):
gcloud compute instances remove-metadata {vm_name} --project={project_id} --zone={zone} --keys=shutdown-script
Case E: VM Lifecycle Events Dashboard & High-Availability SAP Workloads
GCE VM Lifecycle Events MonitoringDashboard (vm-lifecycle-eventsLog-Based Metric): To populate the pre-generatedGCE VM Lifecycle Events Monitoringdashboard in Cloud Monitoring (Monitoring > Dashboards > Dashboard List), create thevm-lifecycle-eventscounter metric using--config-from-fileso the requiredmethodSTRINGlabel extractor onprotoPayload.methodNameis included (gcloud logging metrics create --log-filteralone omits label extractors):# Save as vm-lifecycle-events-metric.yaml: name: vm-lifecycle-events description: "Count of Compute Engine VM lifecycle and host events by method" filter: >- resource.type="gce_instance" AND (log_id("cloudaudit.googleapis.com/activity") OR log_id("cloudaudit.googleapis.com/system_event")) AND operation.first="true" metricDescriptor: metricKind: DELTA valueType: INT64 unit: "1" labels: - key: method valueType: STRING description: "Compute Engine lifecycle or host event method" labelExtractors: method: >- REGEXP_EXTRACT(protoPayload.methodName, "(recreateInstance|hostError|automaticRestart|guestTerminate|terminateOnHostMaintenance|preempted|insert|stop|delete|reset|start)")gcloud logging metrics create vm-lifecycle-events --project={project_id} --config-from-file=vm-lifecycle-events-metric.yaml- SAP Workloads (HANA / NetWeaver):
If the impacted VM runs SAP HANA or SAP NetWeaver, verify automatic
Pacemaker cluster failover to the secondary node in Cloud Monitoring using
the SAP HANA Availability
(
google_agent_for_sap.agent-for-sap-hana-availability-monitoring) or SAP NetWeaver Availability (google_agent_for_sap.agent-for-sap-netweaver-availability-monitoring) dashboards.
Verification, Rollback & Cleanup
- Verify Updated Scheduling Policy:
gcloud compute instances describe {vm_name} --project={project_id} --zone={zone} \ --format="yaml(scheduling, resourceStatus.upcomingMaintenance)" - Remove Test Shutdown Metadata:
gcloud compute instances remove-metadata {vm_name} --project={project_id} --zone={zone} --keys=shutdown-script - Teardown Temporary Test VMs: Delete any temporary test VMs when finished
(omitting
--quieton manual commands so interactive confirmation prevents accidental disk loss):gcloud compute instances delete {vm_name} --project={project_id} --zone={zone} --delete-disks=all - Pricing Reference: Read-only troubleshooting queries create no billable VM resources (user-defined log-based metrics and Log Analytics queries on linked BigQuery datasets follow standard Cloud Monitoring and BigQuery pricing). Consult Google Cloud pricing and Compute Engine pricing for current Standard vs. Spot regional rates rather than hardcoded dollar figures.