RUZIBA TOOLSExplore tools

Robots.txt generator

Create simple crawl directives and a sitemap line.

Stay alertIn development
ScamPause app icon
RUZIBA · IN DEVELOPMENT

ScamPause

A RUZIBA product in development to help people pause before responding to suspicious messages.

Sell simplyAvailable now
SimpleSeller app icon
RUZIBA · ON GOOGLE PLAY

SimpleSeller

Make selling a little simpler. Create listings and keep the details together.

Get it on Google PlayApp Store: coming soon
QR code: scan to open SimpleSeller on Google Play

How the robots.txt generator works

The tool writes a robots.txt file with one User-agent group, a Disallow line for each path you list, and an optional Sitemap line. It builds the text in your browser; nothing is uploaded, and it does not check your live site.

Steps. Enter the User-agent (“*” means all crawlers), then the paths to disallow separated by commas, each starting with “/”. Optionally add the full sitemap URL, which must be http or https. Press Generate robots.txt, review the output in the text box, then press Download to save it as robots.txt.

Example with the default inputs. User-agent “*”, path “/private/” and sitemap https://example.com/sitemap.xml produce: User-agent: *, Disallow: /private/, a blank line, then Sitemap: https://example.com/sitemap.xml. Entering “/admin/, /tmp files/” gives two lines, Disallow: /admin/ and Disallow: /tmp%20files/ (spaces become %20, and # becomes %23). Leaving the path box empty gives a bare Disallow:, which blocks nothing.

Limits. A path that does not start with “/” is rejected, and only a single user-agent can be entered, so rules for several different crawlers need hand editing. There are no Allow lines. Upload the file to the root of the host, for example https://example.com/robots.txt. It is a request that well-behaved crawlers follow, not access control, so do not rely on it to hide private pages. Blocking a path stops crawling, but a blocked URL can still show up in search results if other pages link to it.