robots.txt state could not be established

What this means

The crawl did not obtain a definitive robots.txt for this origin, meaning the request returned 500 or above, timed out, failed to connect, or was refused by the SSRF guard. A plain 404 does not trigger this: per RFC 9309 a 4xx means the file does not exist and no restrictions apply, which the crawler records as a known state. An unavailable file is the opposite case — Googlebot responds by treating the whole site as disallowed and pausing crawling until robots.txt answers again.

How to fix it

Request /robots.txt directly and check the status it returns. Where it answers 5xx or times out, fix the server or the route so it responds — an empty file served with 200, or a clean 404, are both safe outcomes. Make sure robots.txt cannot fail intermittently behind a rate limiter, a WAF, or an application error page.

Fix effort

This check needs a developer fix: it turns on a routing, hosting or infrastructure decision that no automatic edit can make safely.

Authoritative source

https://developers.google.com/search/docs/crawling-indexing/robots/intro