Crawl your site, read the results, ship the fix.
Everything you need to run CrawlX — from your first crawl to the REST API. Paywalled features are marked with their tier.
Getting started
CrawlX is a cloud SEO crawler that finds technical-SEO and AI-readiness issues across your site and — for supported issues — ships the fix as a GitHub pull request. A report tells you what is wrong; CrawlX helps you close the loop.
It is built for two audiences on one platform. SEO teams run crawls, read prioritized issues, generate client-ready reports, and track health over time. Developers drive the same engine through a REST API, signed webhooks, and automated fix PRs.
Run your first crawl
- Create a project for the site you want to audit. A project is one site, identified by its host.
- Start a crawl and choose a mode (spider, sitemap, or URL list — see below). Free projects cover up to 500 URLs per crawl.
- Watch live progress, then review issues grouped by type and ranked by severity, with the highest-leverage fixes first.
From there you can ship fixes, share a report, schedule recurring crawls, or wire CrawlX into your pipeline. Each capability is covered in its own section — paywalled features are marked with their tier.
Crawl modes & scope
Every crawl runs in one of three modes, chosen when you start it:
- Spider (BFS) — the default. Starts at your seed URL and follows internal links breadth-first, up to your depth limit. Sitemaps declared in
robots.txtare merged into the frontier so nothing orphaned is missed. - Sitemap — crawls the URLs listed in your
sitemap.xml(and any sitemaps declared inrobots.txt). Sitemap-index files are followed recursively, up to 50 sitemaps. - URL list — crawls an exact list of URLs you paste in. The list is de-duplicated and clamped to your plan limit.
Scope & limits
Pages per crawl scale with your plan — 500 (Free), 10,000 (Starter), 100,000 (Professional), 500,000 (Agency), and unlimited (Enterprise). Each crawl also has a configurable depth (default 10) and concurrency (default 3 simultaneous requests). See Plans & limits for the full matrix.
How crawling works
CrawlX crawls politely and defensively, so it behaves well against real sites and finishes even when a site pushes back.
- robots.txt is respected by default, including
Allow/Disallowwith wildcards and$anchors, and anyCrawl-delay. - Politeness — requests to the same host are serialized with a minimum delay (a 500 ms floor), and CrawlX backs off further when a host returns 429 or 503.
- Retries — transient failures (a timeout, 429, or 503) get one retry; permanent errors (404, 500) are never retried.
- Stonewall protection — if a site stops responding for many requests in a row, the crawl stops rather than burning your budget against a wall.
Long crawls, live progress, pause & resume
Crawls run in time-boxed segments and checkpoint themselves well before the 300-second serverless ceiling, handing off to a fresh segment — so a 100,000-URL crawl completes across many segments without you doing anything. Progress streams live to the dashboard as pages come in, and you can pause a running crawl and resume it later from the exact checkpoint (its frontier and visited set are preserved).
JavaScript renderingPro+
On Professional and above, CrawlX can render pages in headless Chrome (via Browserless) to see content and links that only appear after JavaScript runs — the same view a modern search engine gets.
- Rendering escalates automatically when a page looks like a bare single-page-app shell (no meaningful content or links in the raw HTML).
- While rendering, CrawlX captures Core Web Vitals (LCP, CLS, FCP, TTFB) via a real
PerformanceObserver. - Limits per crawl segment: up to 40 inline renders, each with a 30-second render timeout. Rendering powers the
render_js_dependentcheck and the rendered-parity factor of AI Readiness.
Reading your results
Every crawl produces a health score from 0–100. Issues are graded into five severities, and the score is weighted so a few criticals matter more than a long tail of minor notes:
- critical — breaks indexing or ranking (e.g. 5xx, missing title, no HTTPS). Weight 10.
- high — meaningful SEO loss (e.g. missing meta description, thin content). Weight 5.
- medium — worth fixing (e.g. canonical issues, slow TTFB). Weight 2.
- low — polish (e.g. generic anchors, un-minified CSS). Weight 0.5.
- info — surfaced for awareness, often intentional (e.g. a noindex directive). No penalty.
For every page, CrawlX records status code, title, meta description, headings, word count, response time, page weight, internal/external links, images, redirect chains, structured-data count, and (when rendered) Core Web Vitals.
How issues are prioritized
Raw findings are noisy, so CrawlX groups and ranks them for you. Issues of the same type are grouped by check, and each group is sorted:
- By severity first — criticals before highs before mediums.
- By reach second — within a severity, the issues affecting the most pages come first.
Each check also carries an impact score (0–100) used to pick the representative example of a group and to drive fix generation. The result: the top of your list is always the highest-leverage work, not just the first thing found.
The SEO check catalog
CrawlX runs 71 checks across 13 categories on every page, on every plan. The full catalog:
HTTP status (4)
| Check | Severity | What it flags |
|---|---|---|
http_4xx | critical | 4xx client errors. 404s are critical; other 4xx are high. |
http_5xx | critical | Server errors (5xx) that block indexing. |
http_redirect | medium | 3xx redirects. 301 is info; 302 (temporary) is flagged medium. |
http_redirect_chain | critical | Redirect chains of 3+ hops that waste crawl budget and dilute link equity. |
Meta data (10)
| Check | Severity | What it flags |
|---|---|---|
meta_title_missing | critical | No <title> tag. |
meta_title_short | medium | Title under 30 characters. |
meta_title_long | low | Title over 60 characters (truncated in SERPs). |
meta_desc_missing | high | No meta description. |
meta_desc_short | low | Meta description under 70 characters. |
meta_desc_long | low | Meta description over 160 characters. |
meta_viewport_missing | high | No viewport tag (mobile-first indexing). |
social_og_missing | low | Missing Open Graph title/description. |
social_og_image_missing | low | Missing og:image. |
social_twitter_card_missing | low | Missing twitter:card. |
Headings (5)
| Check | Severity | What it flags |
|---|---|---|
heading_h1_missing | critical | No H1 on the page. |
heading_h1_multiple | medium | More than one H1. |
heading_h1_short | low | H1 under 10 characters. |
heading_h1_long | low | H1 over 70 characters. |
heading_h2_missing | medium | No H2 subheadings on pages over ~300 words. |
Content (6)
| Check | Severity | What it flags |
|---|---|---|
content_empty | critical | No readable text — often a JS-render problem. |
content_thin | high | Thin content under ~200 words. |
content_title_keywords_missing | medium | Title keywords absent from the body. |
content_keyword_stuffing | high | A keyword over ~3% density / 20+ repetitions. |
duplicate_title | high | The same title on multiple pages (cannibalization). |
duplicate_meta_description | medium | The same meta description on multiple pages. |
Images (4)
| Check | Severity | What it flags |
|---|---|---|
images_alt_missing | high | Images without alt text (high when >5 affected, else medium). |
images_dimensions_missing | medium | Missing width/height (causes layout shift / CLS). |
resource_image_not_optimized | medium | Images not served as WebP/AVIF. |
resource_image_large | high | An image over 1 MB. |
Links (6)
| Check | Severity | What it flags |
|---|---|---|
links_no_internal | high | A page with no internal links. |
links_too_many | medium | Over 150 links on a page (dilutes equity). |
links_high_nofollow_internal | medium | Over half of internal links are nofollow. |
links_empty_anchor | medium | Links with empty anchor text. |
links_generic_anchor | low | Generic anchors like “click here” / “read more”. |
links_external_no_noopener | low | External links missing rel="noopener". |
Indexability & rendering (4)
| Check | Severity | What it flags |
|---|---|---|
index_noindex | info | A noindex directive — surfaced for verification. |
index_canonical_missing | medium | No canonical tag (duplicate-content risk). |
index_canonical_mismatch | medium | Canonical points away from the page URL. |
render_js_dependent | high | Over 30% of content only appears after JavaScript runs. |
Performance & Core Web Vitals (17)
| Check | Severity | What it flags |
|---|---|---|
perf_slow_response | critical | TTFB over 3000 ms. |
perf_moderate_response | medium | TTFB between 1000–3000 ms. |
perf_large_page | high | Page weight over 3 MB. |
perf_moderate_page | medium | Page weight 1–3 MB. |
perf_too_many_scripts | medium | More than 15 JavaScript files. |
render_blocking_critical | high | Over 10 render-blocking resources. |
render_blocking_warning | medium | 5–10 render-blocking resources. |
cwv_lcp_critical | critical | Largest Contentful Paint over 4000 ms. |
cwv_lcp_poor | high | LCP between 2500–4000 ms. |
cwv_cls_critical | critical | Cumulative Layout Shift over 0.25. |
cwv_cls_poor | high | CLS between 0.1–0.25. |
cwv_fcp_critical | high | First Contentful Paint over 3000 ms. |
cwv_fcp_poor | medium | FCP between 1800–3000 ms. |
cwv_ttfb_poor | medium | Time to First Byte over 800 ms. |
resource_css_not_minified | low | Un-minified CSS. |
resource_js_not_minified | low | Un-minified JavaScript. |
resource_oversized | high | Any single resource over 5 MB. |
Security (8)
| Check | Severity | What it flags |
|---|---|---|
security_no_https | critical | Page served over plain HTTP. |
security_no_hsts | medium | Missing Strict-Transport-Security header. |
security_no_xcto | low | Missing X-Content-Type-Options header. |
security_no_csp | low | Missing Content-Security-Policy header. |
security_no_xframe | low | Missing X-Frame-Options (clickjacking). |
security_no_referrer_policy | low | Missing Referrer-Policy header. |
security_no_permissions_policy | info | Missing Permissions-Policy header. |
mixed_content | high | HTTP resources on an HTTPS page (medium when ≤5). |
Structured data (2)
| Check | Severity | What it flags |
|---|---|---|
structured_data_missing | medium | No JSON-LD structured data (blocks rich results). |
structured_data_no_type | high | JSON-LD present but missing @type. |
Internationalization (3)
| Check | Severity | What it flags |
|---|---|---|
hreflang_no_self_reference | medium | Hreflang set missing x-default. |
hreflang_invalid_lang_code | high | Invalid hreflang language/region code. |
hreflang_too_many | medium | Over 50 hreflang entries. |
Pagination (1)
| Check | Severity | What it flags |
|---|---|---|
pagination_missing_prev | medium | A paginated URL without rel=prev context. |
Accessibility (1)
| Check | Severity | What it flags |
|---|---|---|
lang_attribute_missing | low | No lang declaration (html lang / Content-Language). |
AI ReadinessStarter+
AI Readiness scores how accessible your site is to AI crawlers and answer engines — a 0–100 score with an A–F grade, built from five weighted factors:
| Factor | Weight | What it measures |
|---|---|---|
| Rendered parity | 35% | How much content is hidden from non-JS crawlers (raw vs. rendered HTML gap). |
| Bot access | 20% | Share of known AI bots your robots.txt allows. |
| Structured data | 20% | JSON-LD coverage and validity across pages. |
| Semantic structure | 15% | Clean headings and landmarks vs. div-soup (sampled). |
| llms.txt | 10% | Presence and validity of an /llms.txt discovery file. |
The 14 AI bots checked
Bot access reads your robots.txt and reports, per bot, whether the site root is allowed. The current table covers:
- OpenAI — GPTBot, OAI-SearchBot, ChatGPT-User
- Anthropic — ClaudeBot, Claude-Web, anthropic-ai
- Perplexity — PerplexityBot, Perplexity-User
- Google Google-Extended · Apple Applebot-Extended · Amazon Amazonbot · ByteDance Bytespider · Common Crawl CCBot · Meta Meta-ExternalAgent
Scope: this is a robots.txt access check, not log-based bot-traffic analysis. The bot table is dated in-product so you can see when it was last reviewed.
AI fixes & BYOKStarter+
On Starter and above, CrawlX uses AI to draft fixes for supported issues — for example writing a 150–160 character meta description or a concise H1 — which then feed the GitHub pull-request flow below.
Bring your own key (BYOK)
AI runs on your provider key, set per project. Supported providers:
- Anthropic — Claude (Sonnet, Opus, Haiku)
- OpenAI — GPT-4.1 family and GPT-4o
- Google — Gemini 2.5 Pro / Flash and 2.0 Flash
Keys are vaulted server-side — never returned to the browser (only a “set / last-4” hint is shown) and redacted from logs. Each project has a configurable token budget (default 1,000,000 tokens) that is enforced atomically on every call, so a runaway job cannot blow past your cap.
Ship a fix as a pull request
Connect a repository through the CrawlX GitHub App (crawlx-ai), and CrawlX can open a pull request that fixes an issue in your source.
The flow
- Connect — install the GitHub App and pick the repository and default branch for the project (requires the
settings.managepermission). - Preview — CrawlX reads the live repo, resolves the target file (with a confidence score you can override), and shows the exact diff. Nothing is written yet.
- Open PR — on confirm, CrawlX commits to a fresh branch
crawlx/fix-<issue>and opens a PR against your default branch with a traceable body linking back to the crawl and issue.
Guardrails are non-negotiable:
- CrawlX never commits to your default branch and never auto-merges — you review every PR.
- Every generated change is validated before the PR opens: it must parse, be surgical (non-destructive), and actually contain the fix it promised.
- One PR per issue — re-running a fix returns the existing PR instead of opening a duplicate.
meta_desc_missing) and missing H1s (heading_h1_missing) on Next.js source. More fix types are rolling out; every other issue stays a guided manual fix with a suggested change.Reports & exports
Turn a crawl into a shareable report. Five templates are available on every plan:
- Executive Summary — health score, top recommendations, severity mix. One page for stakeholders.
- Technical Audit — full issue list by category, performance distribution, and a page-by-page table.
- Monthly Snapshot — month-over-month deltas with trend arrows.
- Content Audit — thin vs. long-form pages and content issues by impact.
- Competitive Overview — your metrics side-by-side with a competitor crawl.
PDF, white-label & data export
Any report downloads as a real text-based PDF. Starter+ lets you white-label it with your own logo, brand color, and company details — the CrawlX mark disappears once you set any branding. Raw data exports as CSV, JSON, or a full ZIP (crawl + pages + issues), and crawls can export a clean XML sitemap. (There is no XLSX export.)
Sharing
Each report has a read-only public link backed by a secret token. You can set an expiry on the link and rotate the token to revoke the old one. Shared links are rate-limited to prevent enumeration.
Compare crawls & trends
Compare two crawls
Diff any two completed crawls to see exactly what changed between them:
- New issues and fixed issues (matched by check and URL)
- New pages and lost pages
- Status-code changes across URLs
- Health-score delta
Trends over time
Each project has a Trends view charting history over the last 7, 30, 90, or 180 days. Track health score, total issues, critical issues, pages crawled, and average response time, with the change since the start of the window.
SchedulingStarter+
On Starter and above, crawls can run automatically on a daily, weekly, or monthly cadence, at a time and timezone you choose. Each run can notify you and fire a webhook when it completes.
Integrations
Connect data sources per project to enrich crawl results with real demand and performance data. Configure credentials in project settings.
- Google Analytics 4 — sessions, pageviews, and engagement per URL, overlaid on your crawl.
- Google Search Console — impressions, clicks, CTR, and average position per URL, plus top queries.
- PageSpeed Insights — Lighthouse performance, accessibility, best-practices, and SEO scores plus field Core Web Vitals.
- GitHub — the fix-PR delivery covered in Ship a fix as a pull request.
Free tools
Six no-login tools run entirely in your browser — no data leaves the page. Handy for a quick one-off check without starting a crawl.
- Robots.txt Tester — test whether a URL is allowed for a given user-agent.
- Schema Markup Validator — validate JSON-LD and spot missing fields.
- SERP Snippet Preview — preview your title and description in Google results.
- Meta Tag Checker — extract and validate meta, Open Graph, and Twitter tags.
- Hreflang Checker — validate hreflang codes and x-default.
- Heading Structure Checker — view the H1–H6 outline and find gaps.
See them all at /free-tools.
REST APIPro+
On Professional and above, drive CrawlX from your own code. Generate scoped API keys (cx_live_… for production, cx_test_… for sandbox), sent as a bearer token. Keys carry a read or crawl-trigger scope.
# trigger a crawl (idempotent)
curl -X POST https://www.crawlx.ai/api/v1/projects/PROJECT_ID/crawls \
-H "Authorization: Bearer cx_live_…" \
-H "Idempotency-Key: 9d8c7b6a-5e4f-3a2b-1c0d-9e8f7a6b5c4d" \
-H "Content-Type: application/json" \
-d '{"mode":"spider","max_urls":5000}'Endpoints cover the full loop:
- List projects, list and trigger crawls
- Get a crawl’s status and stats
- List a crawl’s issues and pages
- Export a crawl report (JSON or CSV)
Triggering is idempotent via the Idempotency-Key header (a replay returns the original crawl). Requests are rate-limited per key and respond with 429 Too Many Requests plus a Retry-After header. The complete endpoint reference, with request/response examples, lives at /docs/api.
Webhooks
Subscribe a project to events and CrawlX POSTs a signed JSON envelope to your URL when they fire:
crawl.completed·crawl.failedissue.found·fix.pr_opened·schedule.run
Each delivery is signed with the webhook secret. The X-CrawlX-Signature header is t=<unix>,v1=<hmac-sha256> over `${t}.${body}`} — verify on the raw body, compare in constant time, and reject timestamps more than 300 seconds old to defeat replays. The event_id is stable across retries, so use it to dedupe.
Delivery is reliable:
- Failed deliveries retry up to 6 times with full-jitter exponential backoff, then dead-letter.
- You can replay a past delivery from the dashboard.
- Slack incoming-webhook URLs are auto-formatted as Block Kit messages.
- An SSRF guard refuses delivery to private/loopback addresses.
Managing webhooks requires the webhooks.manage permission on the project.
Accounts, teams & retention
Work is organized into projects (one site each). Higher plans add organizations for grouping projects and inviting teammates with role-based access — a viewer can read, an editor can change settings and trigger work.
Project, team-seat, and organization counts are set by your plan (see below). Raw crawl data (pages and issues) is retained per plan — 90 days on Free up to 730 on Agency, and indefinitely on Enterprise — after which it is purged. Aggregated stats, health history, and generated reports are kept regardless.
Plans, limits & billing
Limits and gated features by plan:
| Free | Starter | Professional | Agency | Enterprise | |
|---|---|---|---|---|---|
| Price | $0 | $29/mo | $79/mo | $149/mo | Sales |
| URLs / crawl | 500 | 10,000 | 100,000 | 500,000 | Unlimited |
| Crawls / month | 5 | 25 | 100 | Unlimited | Unlimited |
| Projects | 1 | 5 | 25 | Unlimited | Unlimited |
| Team members | 1 | 1 | 5 | 20 | Unlimited |
| Data retention | 90d | 180d | 365d | 730d | Forever |
| AI features & AI Readiness | — | ✓ | ✓ | ✓ | ✓ |
| Scheduled crawls | — | ✓ | ✓ | ✓ | ✓ |
| White-label reports | — | ✓ | ✓ | ✓ | ✓ |
| JS rendering | — | — | ✓ | ✓ | ✓ |
| Custom extraction | — | — | ✓ | ✓ | ✓ |
| REST API | — | — | ✓ | ✓ | ✓ |
The 71 SEO checks, reports, sharing, exports, compare, and trends are available on every plan, including Free. Webhooks and GitHub fix-PRs are gated by project permission rather than plan. Billing runs through Stripe or Paddle — upgrade, change plan, open the billing portal, or cancel/resume from your account.
Glossary
- Crawl
- One run over a project’s URLs that produces pages, issues, and a health score.
- Frontier
- The queue of discovered-but-not-yet-fetched URLs in a spider crawl.
- Health score
- A 0–100 summary of a crawl, weighted by issue severity.
- AI Readiness
- A 0–100 score for how accessible your site is to AI crawlers and answer engines.
- BYOK
- Bring Your Own Key — AI features run on your own provider API key.
- Guardrail
- The validation a generated fix must pass (parses, surgical, meets expectation) before a PR opens.
- Idempotency key
- A header that makes a repeated API crawl-trigger return the original crawl instead of a duplicate.
- Dead-letter
- A webhook delivery that has exhausted its retries and stopped being attempted.
Start crawling free.
Run your first crawl on up to 500 URLs — no card required.