About our crawler

ShodhyaBot

ShodhyaBot is the crawler of Shodhya, a web search engine being built in India. Until October 2026 it was called TarkaBot, and it still answers to that name: a robots.txt group written for TarkaBot is followed exactly as one written for ShodhyaBot, so nothing you have set needs changing. If it has visited your site, this page says what it was doing and how to control it.

How to recognise it

Every request carries this user agent and a contact address in the From header:

Mozilla/5.0 (compatible; ShodhyaBot/0.1; TarkaBot/0.1; +https://shodhya.com/bot)
From: bot@gettarka.com

Where it comes from

ShodhyaBot's requests come only from the addresses below. Each has reverse DNS under gettarka.com that resolves back to the same address, so a request using the name from anywhere else is not ours.

143.244.143.29    crawl-1.gettarka.com
206.189.143.209   index-1.gettarka.com

The same list, for machines: shodhya.com/bot-ips.json.

What it is for

Pages are fetched to build Shodhya's search index, so that people searching can find them and are sent to the site. They are not used to train AI models and are not sold or passed on.

What it does

• Reads your robots.txt before anything else and follows it. If the file cannot be read because of a server error, it leaves the site alone.
• Makes one request at a time to a site, at least a second apart, and slows further if your server is slow to answer.
• Backs off when it gets a 429 or 503, and waits as long as Retry-After says.
• Fetches HTML pages, a site's icon, and what is said of videos below. It does not submit forms or log in.
• Does not run JavaScript, except on a short list of sites whose pages are empty without it. There it loads the page in a browser under the same name and rules: your robots.txt also governs what the page's scripts ask your site for, and no pictures, fonts or video are fetched.
• For a YouTube video that pages it has read link to, it asks YouTube's oEmbed address for the video's title and channel, and fetches the video's small preview picture, so that a search can show the video with a link to it. One request a second, under the robots.txt of both hosts. If a video or its picture is yours and you want it out of Shodhya, write to the address below.
• For a search for the weather somewhere Shodhya keeps no forecast for, our server asks MET Norway (api.met.no) for that place's forecast, under its robots.txt and terms. The request carries the place's coordinates and nothing else: not the search, not the address of the person searching, no cookie. A forecast is kept until it expires, and at most five places are asked for a second.
• Follows noindex and nofollow, whether in a <meta name="robots"> tag, an X-Robots-Tag header, or a link's rel.

How to control it

To keep ShodhyaBot away from part of your site, or all of it, add a group for it to robots.txt:

User-agent: ShodhyaBot
Disallow: /private/

User-agent: ShodhyaBot
Disallow: /

To slow it down, set a delay in seconds between requests:

User-agent: ShodhyaBot
Crawl-delay: 10

A group for User-agent: TarkaBot is honoured too, and when a file names both, the stricter rule wins for each path. Rules for User-agent: * apply when there is no group naming either. Changes are picked up within 24 hours.

How to reach us

• Questions about the crawler, requests to slow down or stop, and requests to remove pages: bot@gettarka.com
• Reports of network abuse: abuse@gettarka.com

Please include your domain and, if you can, a few lines from your server log. We read both inboxes.

ShodhyaBot/0.1 (formerly TarkaBot) · shodhya.com