Krawly.io is a free online platform that provides 160+ web scraping, SEO analysis, security auditing, and developer tools. It allows users to extract data from websites, analyze SEO performance, check security headers, detect technologies, scrape YouTube data, generate leads, and much more — all without requiring signup or installation.

Is Krawly free to use?

Yes! You get 10 free uses per week without signing up. Create a free account to get 100 credits as a signup bonus, or upgrade to a paid plan starting at $9/month for 1,000 credits.

Do I need to install anything?

No installation needed. All 160+ tools run directly in your browser with no downloads or setup required. There is also a REST API and a Chrome extension for advanced users.

What data formats can I export?

All tools support JSON, CSV, and Excel (XLS) export. The screenshot tool provides PNG images. API responses are in JSON format.

How does the Krawly API work?

After signing up, you get an API key from your dashboard. The REST API accepts JSON POST requests and returns structured data. All 160+ tools are available via the API at https://krawly.io/api/v1/tools/.

What types of tools does Krawly offer?

Krawly offers tools in 11 categories: SEO (meta tag validator, heading analyzer, keyword density, core web vitals), Web Scraping (email scraper, table extractor, CSS selector scraper), Security (SSL checker, port scanner, WordPress scanner), OSINT (DNS lookup, WHOIS, IP geolocation), E-Commerce (Amazon scraper, Shopify scraper, price tracker), YouTube (comments scraper, playlist extractor, thumbnail downloader), Developer (JSON formatter, regex tester, JWT decoder), Content (word counter, readability score), Analysis (technology detector, page speed analyzer), Social Media (TikTok analyzer, Instagram analyzer), and Utilities (QR code generator, URL shortener).

Is Krawly a good alternative to SEMrush, Ahrefs, or BuiltWith?

Krawly provides many of the same on-page SEO analysis features as SEMrush and Ahrefs for free, including meta tag validation, heading analysis, keyword density checking, and structured data auditing. For technology detection, it serves as a free alternative to BuiltWith and Wappalyzer. However, Krawly focuses on individual page analysis rather than domain-wide crawling or backlink databases.

Can I use Krawly for lead generation?

Yes. Krawly's Lead Generation tool automatically crawls a website's homepage, about, contact, and team pages to extract email addresses, phone numbers, LinkedIn profiles, Twitter handles, Facebook pages, Instagram accounts, and GitHub links. Results can be exported as JSON, CSV, or Excel.

How does robots.txt work?

Robots.txt tells search engine crawlers which URLs they can access. It uses User-agent, Allow, and Disallow directives.

Back to all tools

Robots.txt Tester

SEO

Fetch and analyze robots.txt rules. Test which paths are allowed or blocked for crawlers.

Unlock Full Power with Krawly Pro

Get access to all 150+ tools with higher limits. Start with 100 free credits — no credit card required.

Use via API

Robots.txt Tester — cURL

curl -X POST "https://krawly.io/api/v1/tools/robots-txt-tester/" \
  -H "Content-Type: application/json" \
  -H "Authorization: Bearer YOUR_API_KEY" \
  -d '{"url": "https://example.com"}'

Get 100 Free Credits View Pricing Plans API Docs

150+ Tools Full API Access Bulk Processing Priority Support

What is Robots.txt Tester?

Robots.txt Tester fetches and parses the robots.txt file from any domain. It shows all rules, sitemaps, user agents, and tests common paths to show whether they're allowed or blocked.