API + MCP — manage indexing
from IDE, AI agent, or CI/CD
REST API for scripts. MCP for Claude, Cursor, Windsurf. One Bearer token — full indexing automation for technical teams.
No sitemap • Quick Start
Key features
Everything you need to speed up your site's indexing
REST API v1
7 endpoints, JSON, 60 req/min. POST /submit — up to 100 URL per request. curl, Python, Node.js.
MCP for AI agents
8 tools: list_projects, list_sites, submit_url, submit_urls, get_status, get_history and others. Streamable HTTP.
Unified Security
MCP is proxied through REST API v1. Same rate limits, plan limits, audit trail.
AI Content Pipeline
Claude generates content → MCP submit_urls → indexing. Full AI-to-index pipeline.
CI/CD integration
git push → build → deploy → API call → indexing. GitHub Actions, Jenkins, GitLab CI.
CMS webhooks · Soon
Crawlero-native CMS modules are being rebuilt. The REST API remains available for custom workflows.