← LeaderboardThe Skill Leaderboard

gke-observability

Configures GKE observability, including Cloud Logging, Cloud Monitoring, and managed Prometheus. Use when configuring GKE monitoring, setting up GKE logging, or configuring Prometheus metrics collection, to diagnose GKE control plane, API server, admission webhook, etcd storage, node registration or label, and pod eviction, preemption, or scheduling issues using GKE metrics, Kubernetes events, and Cloud Logging, and to troubleshoot Managed Service for Prometheus (GMP) issues such as missing metrics, unhealthy scrape targets, PodMonitoring misconfiguration, rule/alert evaluation failures, and m

Published by google ↗google/skills

Try your first task

A source-grounded starter task has not been verified for this skill yet.

SKILL.md

name: gke-observabilitydescription: >-  Configures GKE observability, including Cloud Logging, Cloud Monitoring, and  managed Prometheus. Use when configuring GKE monitoring, setting up GKE  logging, or configuring Prometheus metrics collection, to diagnose GKE control  plane, API server, admission webhook, etcd storage, node registration or  label, and pod eviction, preemption, or scheduling issues using GKE metrics,  Kubernetes events, and Cloud Logging, and to troubleshoot Managed Service for  Prometheus (GMP) issues such as missing metrics, unhealthy scrape targets,  PodMonitoring misconfiguration, rule/alert evaluation failures, and monitoring  permission errors. Don't use to configure local application logging frameworks  or external APMs outside GKE.metadata:   version: "1.2.1"  category: CloudObservabilityAndMonitoring

GKE Observability

This reference covers monitoring, logging, and metrics configuration and troubleshooting for Google Kubernetes Engine (GKE). The golden path enables comprehensive observability including control-plane metrics and Managed Service for Prometheus (GMP).

MCP Tools: get_cluster, list_k8s_events, get_k8s_logs, get_k8s_cluster_info, describe_k8s_resource. CLI-only: gcloud container clusters update --monitoring=..., gcloud logging read

Usage Attribution

Do not run live kubectl, gcloud, or curl commands unless the user names a target cluster/project or asks to inspect a live cluster (output commands and queries with placeholders, and skip Usage Attribution). Before running any live command, read Usage Attribution, prefix gcloud with CLOUDSDK_METRICS_REQUEST_ATTRIBUTION="agent-skills/1.0 (skill:gke-observability)", and append agent-skills/1.0 (skill:gke-observability) to User-Agent headers (never in generated scripts, Terraform, or user-facing code).

Sub-Skill Reference Routing (Progressive Disclosure)

Complete Reference Index (no scripts/ directory or unlisted reference files exist — do not run ls on the skill folder; read only the matching reference below when answering deep diagnostic, metric, log, or GMP questions):

Control-Plane Metrics, Storage, Webhooks & Capacity: Read for GKE control-plane (KCP) VM CPU or RAM/memory usage questions (not publicly exposed) and customer-visible alternatives; cluster etcd / etcd-compatible storage left, quota & per-resource scalability limits; admission webhook latency, system namespace/Lease/upgrade exclusions, APIService discovery bloat, client discovery caching bursts (/apis/custom.metrics.k8s.io, /apis/metrics.k8s.io) & proactive GKE Recommender insights; watch storms; controller-manager workqueue & scheduler lag; or control-plane headroom, VM autoscaling & node-count pre-scaling, network egress, connections (65k, no per-replica metric) & Protobuf.
Cloud Logging, Audit Attribution, APF & API Server Traces: Read for kube-apiserver slow-request Trace[ logs (trace.go & KEP-647); httplog.go client load (user_agent, verb, apf_iseats); Cloud Audit Log noisy-neighbor, heavy LIST & unindexed CRD fieldSelector scan tail-latency attribution; API Priority and Fairness (APF) 429/504/404 throttling; 409 Conflict hot objects (ConfigMap -> Lease heartbeats) & RenewDeadline / 2 leader-election timeouts; pods/exec saturation, Konnectivity (No agent available), disabled Cloud KMS keys, or in-cluster GCP IAM OAuth token webhook authorizer saturation.
Cluster/Node/Pod Health, Maintenance & Managed Prometheus (GMP): Read for control-plane availability (kubernetes.io/cluster/kcp_healthchecks rate ratio & ruling out KCP outages); maintenance windows (in_maintenance_window), zonal resize downtime & cluster access errors (EOF / masterAuthorizedNetworksConfig); node readiness, registration (AUTO_REPAIR_NODES, Private Google Access) & persistent node-pool labels; pod startup timelines, crashing pod logs, tolerationSeconds: 0 mass eviction & Autopilot preemption; or multi-burn-rate SLO alerting (GlobalRules) & GMP / Grafana / PodMonitoring manifests & troubleshooting.

Golden Path Observability Defaults

Setting	Golden Path Value	Notes
`loggingConfig` components	SYSTEM_COMPONENTS, WORKLOADS	Full workload logging
`monitoringConfig` components	SYSTEM_COMPONENTS, STORAGE, POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, JOBSET, CADVISOR, KUBELET, DCGM, APISERVER, SCHEDULER, CONTROLLER_MANAGER	Full suite including control-plane
`managedPrometheusConfig.enabled`	`true`	Google-managed Prometheus
`advancedDatapathObservabilityConfig.enableMetrics`	`true`	Dataplane V2 flow metrics
`loggingService`	`logging.googleapis.com/kubernetes`	Cloud Logging
`monitoringService`	`monitoring.googleapis.com/kubernetes`	Cloud Monitoring

Enabling Full Monitoring & Logging

Include all four caveats whenever providing a --monitoring or --logging command so users avoid accidental metric loss, surprise billing, or silent IAM failures:

Control-plane metrics and logs are NOT enabled by default. State this outright: API_SERVER, SCHEDULER, and CONTROLLER_MANAGER are off on every new cluster (as are DCGM, CADVISOR, KUBELET, and kube-state POD, DEPLOYMENT, STATEFULSET, DAEMONSET, HPA, STORAGE, JOBSET). Only SYSTEM is enabled by default. Link to https://docs.cloud.google.com/kubernetes-engine/docs/how-to/control-plane-metrics and https://docs.cloud.google.com/kubernetes-engine/docs/how-to/view-logs#control_plane_logs.
The flag replaces, it does not append. Passing --monitoring or --logging overrides the existing component list; pass the full desired list including SYSTEM.
Control-plane and kube-state metrics bill per sample ingested via Managed Service for Prometheus (GMP). Mention the cost impact on large clusters; use --monitoring=SYSTEM in non-production to reduce cost.
Verify APIs and IAM permissions (PermissionDenied). Check gcloud container clusters describe {cluster_name} --region {region} --format="yaml(loggingConfig,monitoringConfig)", ensure logging.googleapis.com and monitoring.googleapis.com are enabled, and verify the node service account / GKE service agent has roles/monitoring.metricWriter and roles/container.defaultNodeServiceAgent.

CLI flag vs. API enum spelling (mixing them fails the command):

Component gcloud --monitoring= monitoringConfig API enum
System SYSTEM SYSTEM_COMPONENTS
API server API_SERVER APISERVER
Controller mgr CONTROLLER_MANAGER CONTROLLER_MANAGER

Component	`gcloud --monitoring=`	`monitoringConfig` API enum
System	`SYSTEM`	`SYSTEM_COMPONENTS`
API server	`API_SERVER`	`APISERVER`
Controller mgr	`CONTROLLER_MANAGER`	`CONTROLLER_MANAGER`

# Enable golden path monitoring and control-plane logging suite
gcloud container clusters update {cluster_name} --region {region} \
  --monitoring=SYSTEM,API_SERVER,SCHEDULER,CONTROLLER_MANAGER,STORAGE,POD,DEPLOYMENT,STATEFULSET,DAEMONSET,HPA,JOBSET,CADVISOR,KUBELET,DCGM \
  --logging=SYSTEM,WORKLOAD,API_SERVER,SCHEDULER,CONTROLLER_MANAGER \
  --quiet

# Enable Managed Prometheus and Dataplane V2 flow/policy observability
gcloud container clusters update {cluster_name} --region {region} \
  --enable-managed-prometheus \
  --enable-dataplane-v2-flow-observability \
  --quiet

Core GKE Metrics & Alerting Baseline

Explicitly name Google Cloud Monitoring when proposing dashboards and alerts, and cover cluster/node CPU and memory capacity, pod restarts (CrashLoopBackOff), volume space exhaustion, and API server latency (apiserver_request_duration_seconds). Direct control-plane VM CPU/RAM metrics are not publicly exposed — read Control-Plane Metrics, Storage, Webhooks & Capacity whenever asked about control-plane VM CPU/memory or deep control-plane metrics.

Metric	Source	Use / Alert Threshold
`apiserver_request_duration_seconds`	API Server (GMP)	Control-plane latency (Alert: P99 > 5s)
`scheduler_scheduling_attempt_duration_seconds`	Scheduler (GMP)	Current GKE scheduling latency (`scheduler_schedule_attempts_total{result="error"} > 0`)
`container_cpu_usage_seconds_total`	cAdvisor (Node)	Worker-node container CPU usage
`container_memory_working_set_bytes`	cAdvisor (Node)	Worker-node container memory usage
`kubernetes.io/node/cpu/core_usage_time`	Cloud Monitoring	Node CPU usage (`kubernetes.io/node/memory/used_bytes` for memory)
`kube_pod_status_phase`	kube-state-metrics	Pod lifecycle phase (requires deploying GMP or self-managed kube-state-metrics)
`kube_pod_container_status_restarts_total`	kube-state-metrics	Pod crash loops (Alert: > 5 in 10m; requires kube-state-metrics)
`kube_node_status_condition`	kube-state-metrics	Node readiness (`condition="Ready", status!="True"`; or `kubernetes.io/node/status_condition`)
`kubelet_volume_stats_used_bytes / capacity`	Kubelet	PVC capacity utilization (Alert: > 85%)
`DCGM_FI_DEV_GPU_UTIL`	DCGM (GMP)	GPU utilization (Alert: > 95% sustained)

Live Resource Usage & Cloud Logging Queries

Use kubectl top for live usage (no MCP or gcloud equivalent):

kubectl top pods --all-namespaces --sort-by=cpu
kubectl top nodes
kubectl top pods --containers -n {namespace}

Use gcloud logging read (or Cloud Logging Log Explorer) with LQL filters:

# Control-plane component logs (requires API_SERVER / SCHEDULER / CONTROLLER_MANAGER enabled)
gcloud logging read 'resource.type="k8s_control_plane_component" AND resource.labels.cluster_name="{cluster_name}"' --project {project_id} --limit 50 --quiet

# Container errors: resource.type="k8s_container" AND resource.labels.container_name="{container_name}" AND severity>=ERROR
# OOMKilled events: resource.type="k8s_event" AND jsonPayload.reason="OOMKilling"
# FailedScheduling: resource.type="k8s_event" AND jsonPayload.reason="FailedScheduling"
# Cloud Audit Logs: resource.type="k8s_cluster" AND logName:"cloudaudit.googleapis.com"
# Dataplane V2 network policy / packet drop logs: logName:"policy_action" AND jsonPayload.disposition="DENY"

Distributed Tracing, Profiling & Recent Additions

Cloud Trace & OpenTelemetry: Deploy the OpenTelemetry Collector (googlecloud exporter with Workload Identity) or enable Managed OpenTelemetry for GKE (Preview) on GKE 1.34.1-gke.2178000+ (gcloud beta container clusters update {cluster_name} --region {region} --managed-otel-scope=COLLECTION_AND_INSTRUMENTATION_COMPONENTS).
Cloud Profiler: Continuous low-overhead production CPU/memory profiling.
PSI (Pressure Stall Information) metrics: cAdvisor container_pressure_{cpu,memory,io}_{waiting,stalled}_seconds_total via GMP ClusterNodeMonitoring (GKE 1.35+).