gcloud-usage

name: gcloud-usagedescription: This skill should be used when user asks about "GCloud logs", "Cloud Logging queries", "Google Cloud metrics", "GCP observability", "trace analysis", or "debugging production issues on GCP".

GCP Observability Best Practices

Structured Logging

JSON Log Format

Use structured JSON logging for better queryability:

{
  "severity": "ERROR",
  "message": "Payment failed",
  "httpRequest": { "requestMethod": "POST", "requestUrl": "/api/payment" },
  "labels": { "user_id": "123", "transaction_id": "abc" },
  "timestamp": "2025-01-15T10:30:00Z"
}

Severity Levels

Use appropriate severity for filtering:

• DEBUG: Detailed diagnostic info
• INFO: Normal operations, milestones
• NOTICE: Normal but significant events
• WARNING: Potential issues, degraded performance
• ERROR: Failures that don't stop the service
• CRITICAL: Failures requiring immediate action
• ALERT: Person must take action immediately
• EMERGENCY: System is unusable

Log Filtering Queries

Common Filters

# By severity
severity >= WARNING

# By resource
resource.type="cloud_run_revision"
resource.labels.service_name="my-service"

# By time
timestamp >= "2025-01-15T00:00:00Z"

# By text content
textPayload =~ "error.*timeout"

# By JSON field
jsonPayload.user_id = "123"

# Combined
severity >= ERROR AND resource.labels.service_name="api"

Advanced Queries

# Regex matching
textPayload =~ "status=[45][0-9]{2}"

# Substring search
textPayload : "connection refused"

# Multiple values
severity = (ERROR OR CRITICAL)

Metrics vs Logs vs Traces

When to Use Each

Metrics: Aggregated numeric data over time

• Request counts, latency percentiles
• Resource utilization (CPU, memory)
• Business KPIs (orders/minute)

Logs: Detailed event records

• Error details and stack traces
• Audit trails
• Debugging specific requests

Traces: Request flow across services

• Latency breakdown by service
• Identifying bottlenecks
• Distributed system debugging

Alert Policy Design

Alert Best Practices

• Avoid alert fatigue: Only alert on actionable issues
• Use multi-condition alerts: Reduce noise from transient spikes
• Set appropriate windows: 5-15 min for most metrics
• Include runbook links: Help responders act quickly

Common Alert Patterns

Error rate:

• Condition: Error rate > 1% for 5 minutes
• Good for: Service health monitoring

Latency:

• Condition: P99 latency > 2s for 10 minutes
• Good for: Performance degradation detection

Resource exhaustion:

• Condition: Memory > 90% for 5 minutes
• Good for: Capacity planning triggers

Cost Optimization

Reducing Log Costs

• Exclusion filters: Drop verbose logs at ingestion
• Sampling: Log only percentage of high-volume events
• Shorter retention: Reduce default 30-day retention
• Downgrade logs: Route to cheaper storage buckets

Exclusion Filter Examples

# Exclude health checks
resource.type="cloud_run_revision" AND httpRequest.requestUrl="/health"

# Exclude debug logs in production
severity = DEBUG

Debugging Workflow

1. Start with metrics: Identify when issues started
2. Correlate with logs: Filter logs around problem time
3. Use traces: Follow specific requests across services
4. Check resource logs: Look for infrastructure issues
5. Compare baselines: Check against known-good periods