api-gateway-troubleshooting

name: api-gateway-troubleshootingmetadata:   version: "1.0.0"  category: Serverlessdescription: >-  Diagnoses failing requests and deployment errors on Google Cloud API Gateway.  Use when calls through a gateway return 4xx or 5xx errors, including 429 and  other quota errors, time out, or fail a CORS preflight; when  `api-configs create` fails with a service account, concurrency, or limit  error, or `gateways create` or `gateways update` fails; when a custom domain,  load balancer, or TLS finding involves a gateway; or when the user asks if  Google is at fault. Don't use for Apigee, Cloud Endpoints, or GKE Gateway API.  For OpenAPI errors from `api-configs create`, use  api-gateway-openapi-spec-authoring; for setting up or changing API keys,  JWTs, or quotas, use api-gateway-backend-security-and-rate-limiting; for MCP  `/mcp` errors, use api-gateway-mcp-server.

API Gateway Troubleshooting

Use this skill to find out why requests to a deployed Google Cloud API Gateway fail and how to fix them. Work only from three sources of evidence:

1. Public Google Cloud documentation, linked from this skill.
2. The user's own project, read with the user's own credentials.
3. Files and output that the user provides, such as an OpenAPI document, an error response, or a pasted log entry.

Scope

This skill covers only Google Cloud API Gateway. If the question is about Apigee, Cloud Endpoints, or GKE Gateway API, reply that this skill covers only Google Cloud API Gateway, and don't run its steps or commands.

Usage Attribution

MUST READ: Before executing any gcloud, REST API, or MCP command, you MUST read and follow the instructions in Usage Attribution.

Ground rules

Prerequisites

1. Google Cloud CLI. Use a recent gcloud (the commands here were checked with version 591.0.0); see Install the gcloud CLI. Don't install or update software without the user's approval. If gcloud isn't available or the user says not to run commands, work in evidence mode: give the user the read-only commands to run, and diagnose from the output, response bodies, and log entries they paste.
2. Authentication. Use the user's existing gcloud auth login session; this skill doesn't need application-default credentials.
3. Project. Ask which project hosts the gateway and pass --project={project_id} on every command. Don't run gcloud config set, because it changes the user's environment. This skill troubleshoots existing gateways and never creates projects.
4. Billing. The project needs an active billing account even when usage stays in the free tier; see API Gateway pricing.
5. APIs. API Gateway requires apigateway.googleapis.com, servicemanagement.googleapis.com, and servicecontrol.googleapis.com. A gateway can keep responding while one of them is missing, and its logs then stay empty, so check with gcloud services list --enabled --project={project_id}. Enabling a missing API is a change that needs approval.
6. IAM roles for diagnosis (read-only): roles/apigateway.viewer, roles/logging.viewer, roles/serviceusage.serviceUsageViewer, and roles/iam.serviceAccountViewer (to check the gateway's service account). For Cloud Run backends, add roles/run.viewer; to trace a custom domain through its load balancer, add roles/compute.viewer; to check an API key's project, roles/serviceusage.serviceUsageViewer is also needed there.
7. IAM roles for fixes (only after approval): roles/apigateway.admin; roles/iam.serviceAccountUser on the gateway's service account, because creating or updating an API config or a gateway requires iam.serviceAccounts.actAs; roles/run.admin to change a Cloud Run service's IAM policy or ingress; roles/serviceusage.serviceUsageAdmin to enable an API or a managed service, plus roles/servicemanagement.serviceConsumer on the API's managed service to enable it in another project; and roles/iam.serviceAccountAdmin to re-enable a disabled service account.
8. Region. Gateways are regional. Use the region that gateways list reports. When a fix needs a new gateway, create it in the original gateway's region, or in us-central1 if the user has no preference; see API Gateway deployment model and Cloud locations.

Workflow

Copy this checklist into your notes and tick it off as you go:

- [ ] 1. Context collected
- [ ] 2. Gateway identified by defaultHostname
- [ ] 3. Expected or transient states ruled out
- [ ] 4. Gateway log entry read (bounded query)
- [ ] 5. One branch chosen from the classification table
- [ ] 6. Root cause stated with redacted evidence and a doc link
- [ ] 7. Fix proposed, approved, and applied (or handed to the user)
- [ ] 8. Fix verified
- [ ] 9. Escalated, if the evidence points to Google Cloud

1. Collect context

Extract these from the conversation first, and ask only for what's missing:

If the project or the host is unknown, stop and ask before you run any gcloud command. Don't guess project IDs, and don't use the active gcloud project instead.

2. Identify the gateway

List the project's gateways:

gcloud api-gateway gateways list --project={project_id} \
  --format="table(name.basename():label=GATEWAY,name.segment(3):label=REGION,defaultHostname,state,apiConfig)"

Match the request host against defaultHostname. Don't decide from the URL suffix: a gateway's hostname can use either of two gateway.dev formats or, for older streaming and model-routing gateways, a run.app format. Don't guess the owning project from numbers in the hostname either; list the gateways in the project the user names. A custom domain points at a load balancer whose serverless NEG targets the gateway. See references/triage-and-logs.md for the hostname formats and how to trace a custom domain.

If no gateway in the project matches, confirm the product before going on (see Scope).

Then read the gateway's details, including its API config and streaming mode:

gcloud api-gateway gateways describe {gateway_id} \
  --location={region} --project={project_id}

3. Rule out expected and transient states

• Gateway state isn't ACTIVE: CREATING or UPDATING means an operation is still running, so wait and check again. FAILED means creation failed; read the error of the failed operation. DELETING means the gateway is being removed.
• API config just created: creating a complex API config can take up to ten minutes, and only one config can be created for an API at a time.
• First requests after an idle period: API Gateway has cold-start latency.
• Google Cloud incident: check the Service Health dashboard.

4. Read the gateway log entry

Use a bounded query that prints only the fields you need. Replace {status_code} with the status the caller saw:

gcloud logging read 'resource.type="apigateway.googleapis.com/Gateway" AND resource.labels.gateway_id="{gateway_id}" AND resource.labels.location="{region}" AND httpRequest.status={status_code}' \
  --project={project_id} --freshness=1h --limit=20 \
  --format="json(timestamp,httpRequest.status,httpRequest.latency,httpRequest.requestUrl,jsonPayload.responseDetails,jsonPayload.apiConfig,jsonPayload.apiMethod)"

If you know the failure time, replace --freshness with a timestamp>="{start_utc}" AND timestamp<="{end_utc}" filter that spans about 30 minutes around it. Read httpRequest.status, httpRequest.requestUrl, jsonPayload.responseDetails, jsonPayload.apiConfig, and jsonPayload.apiMethod. Keep the --format field list, because a full entry can contain the caller's credentials, and redact credentials before you quote any field, including a key query parameter in httpRequest.requestUrl.

To compare API configs, compare only the apis/{api_id}/configs/{config_id} part: the same config can appear as a full resource name (//apigateway.googleapis.com/projects/...) and with either the project number or the project ID.

jsonPayload.responseDetails decides the branch: via_upstream means the backend produced the response; any other value means the gateway produced it. The troubleshooting page explains via_upstream and service_control_quota_error, and the model routing page explains the model_router_* values that model-routing gateways log. Quote any other value as-is, and don't explain how the gateway works internally.

5. Classify and follow one branch

Pick the first row that matches, and load only the reference file it names.

If the request also needs log queries or backend inspection commands, use triage-and-logs.md.

6. State the root cause

Give the cause, the evidence (redacted, quoted verbatim from logs or command output), and the public doc section that explains it. If the evidence fits more than one cause, list at most three, say what would tell them apart, and collect that evidence before proposing a fix.

7. Propose the fix

Show the exact command, the resource it changes, its effect on live traffic, and how to undo it. Wait for approval, then run it, or hand it to the user if they prefer. Keep these rules in mind:

• To change the OpenAPI document, create a new API config and deploy it with gcloud api-gateway gateways update. Before you create the config, check quota metrics with the api-gateway-backend-security-and-rate-limiting skill, because the newest API config's quota metrics apply to every gateway of the API.
• Before you create a new API config, check the OpenAPI document with the linter (see OpenAPI pre-flight check).

8. Verify

Read the logs again for new entries after the change, or repeat the failing request. Repeat a request yourself only if it's safe to repeat, such as a GET; otherwise ask the user to repeat it, because a repeated POST can change data in their backend. After gateways update, wait until the gateway's state is ACTIVE before you test.

9. Escalate when the evidence points to Google Cloud

Escalate to Cloud Customer Care when the service account is active, the required APIs are enabled, and no log exclusion or wrong scope explains it, but requests get 500 errors with no gateway log entry; when several gateways fail at once without a config change; when errors are intermittent on an unchanged config while the backend logs stay clean; when control-plane operations keep failing with internal errors; or when Service Health reports an incident. Draft the case text with the evidence bundle in limits-and-escalation.md. The user files the case.

OpenAPI pre-flight check

Run the linter on every OpenAPI file before creating an API config. Run it; don't read its source. If you don't have the file, give the user the command to run:

python3 scripts/lint_openapi_for_gateway.py {spec_path} [{spec_path} ...]

It reads the files only, makes no network calls, and prints JSON findings that each link to the public doc for the limitation. Exit code 0 means no errors, 1 means at least one ERROR finding, and 2 means a usage, parse, or dependency error. The linter needs Python 3.8 or later. YAML input needs PyYAML; ask the user before you install it with pip install "PyYAML>=6.0,<7.0". JSON input doesn't. Fix the ERROR findings, and then create the API config. If api-configs create still fails on the OpenAPI document, hand off to api-gateway-openapi-spec-authoring, or use the error message if that skill isn't installed. Show WARNING findings to the user and explain them.

Output format

Report findings in this form:

**Problem:** <one sentence in the user's terms>
**Failed hop:** <gateway or backend, and the evidence for it>
**Evidence:** <redacted log fields or command output, verbatim>
**Root cause:** <one or two sentences> (<public doc link>)
**Fix:** <exact command or spec change; approval needed: yes/no>
**Verify:** <how to confirm the fix>
**Open questions:** <anything you couldn't confirm>

Related skills

Reference files

• references/attribution.md: usage attribution rules for every command you run.
• references/triage-and-logs.md: hostname formats, log queries, log fields, metrics, and backend inspection commands.
• references/auth-errors.md: 401 and 403 on either hop, API keys, JWTs, and the gateway service account.
• references/backend-routing-and-timeouts.md: 404, size limits, backend 5xx, 504 and deadlines, CORS, and streaming.
• references/config-deployment-errors.md: API config and gateway create, update, and delete failures other than OpenAPI errors, and quota metric conflicts.
• references/networking-domains-tls.md: custom domains, load balancers, TLS, and requests that never reach the gateway.
• references/limits-and-escalation.md: product limits, quota increases, unsupported features, and the Cloud Customer Care evidence bundle.
• scripts/lint_openapi_for_gateway.py: offline linter for documented OpenAPI limitations.