» Home » Robots » Page 1
django: trying to access my robots.txt: “TypeError at /robots.txt 'str' object is not callable”
by darko.topolsek in Programming Languages

Exception Type: TypeError at /robots.txt

Exception Value: 'str' object is not callable

What gives?


ROBOTS_PATH = os.path.join(CURRENT_PATH, 'robots.txt')
def robots(request):
""" view for robots.txt file """
return HttpResponse(open(ROBOTS_PATH).read(), 'text/plain')



How to stop robots crawling pagination using robots.txt?
by turret in Development Tools & Services

I have various paginations on my site and I want to stop google and other search engines crawling the index of my paginations.

Example of a crawled page:


How can I, using robots.txt deny robots from crawling any pages that contain /index/ ?

Robots are still disallowed even without a robots.txt file
by KeithTalent in Development Tools & Services

I'm having trouble with a website I'm working on. I initially set up a robots.txt file to prevent robots from indexing it while I was working on it. However now its live and the robots.txt file has been deleted but it still has not been crawled and shows that robots are disallowed access. Even in the absence of a robots.txt file. The site is a wordpress ba

No Robots robots.txt Location
by Kuer in Operating Systems

A bit confused with robots.txt.

Say I wanted to block robots on a site on a Linux based Apache server in location:


I would place robots.txt in that directory (alongside index.php) containing this:

User-agent: *
Disallow: /


Does that stop robots indexing the whole server or

Redirect robots to robots-beta.txt when the beta.* prefix is being used
by slippysoup in Development Tools & Services

I'm trying to disable all indexing on all content coming from my beta.* sub-domain. I don't actually have a beta/ directory on the server. The app is smart enough to switch the user to the beta code when it sees the beta prefix in the url.

Anyway, what I'm trying to do is redirect all robots to robots-beta.txt when the beta.* prefix is being used, but I'm having a hard time getting

War Robots
by Mike Bakalov in Computers
War robots have a relatively brief history in combat. World War II saw some of the first developments, but it wasn't until the United States launched its wars against Afghanistan and Iraq that combat robots became pivotal to the battlefield. Weapons like the Predator and the Foster-Miller TALON have shown their efficiency and importance in battle. However, the future of combat robots is still bein
TAGS : Robots

Robots.txt in ASP.NET MVC
by Mark in ASP & ASP.net

I am trying to figure out what to add to my robots.txt file ? Specifically, what does the command

Allow: /$

do in the robots.txt file ?

Edit: Also, how to allow a site to any have its /index page indexed when using ASP.NET MVC ?

TAGS : Robots

How to Get Rid of Robots.txt
by cmt95 in Internet
A robots.txt file can prevent search engine crawlers from accessing your entire website or parts of it and adding it to the index. If you do not wish to block your website from appearing in search results anymore, you delete the robots.txt file to allow crawlers to index your website again.Difficulty:Moderately EasyInstructions Open your Internet browser and connect to your Web host's file mana
TAGS : Robots

robots.txt and seo ranking
by ondrej in Development Tools & Services

is there any point in having
a robots.txt with just one line of content like below

User-agent: *

compared to not having it at all ?

TAGS : robots ranking

can I get a explanation about robots.txt
by Roman in Programming Languages

I just want to know whether you need robots.txt in your website for the crawler to index the pages. Or it is just to disallow any content that you dont want to index.

Privacy Policy - Copyrights Notice - Feedback - Report Violation - RSS 2017 © bighow.org All Rights Reserved .