Return a different robots.txt for a given hostname
3 posts
• Page 1 of 1
Return a different robots.txt for a given hostname
Hello,
Is it possible to return a different robots.txt file if a certain hostname is used to request the file? (Use case: I want to send an "ignore all" type of robots.txt for one of the hostnames registered with my application, but a standard one for all others).
So something like (excuse horrible pseudo code):
Given a request for "robots.txt"
If hostname = "cdn.foo.com" Then
Return "robots-cdn.txt"
Else
Return "robots.txt"
End If
I'm assuming I'd want to do this with something like mod_rewrite, but am open to any suggestions.
Thanks in advance!
Is it possible to return a different robots.txt file if a certain hostname is used to request the file? (Use case: I want to send an "ignore all" type of robots.txt for one of the hostnames registered with my application, but a standard one for all others).
So something like (excuse horrible pseudo code):
Given a request for "robots.txt"
If hostname = "cdn.foo.com" Then
Return "robots-cdn.txt"
Else
Return "robots.txt"
End If
I'm assuming I'd want to do this with something like mod_rewrite, but am open to any suggestions.
Thanks in advance!
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Return a different robots.txt for a given hostname
Hello,
The idea is to use mod_rewrite. Use httpd.conf:
You can learn more from examples section
Regards
Andrew
The idea is to use mod_rewrite. Use httpd.conf:
- Code: Select all
RewriteEngine on
RewriteCond %{HTTP:Host} ^cdn\.foo\.com$ [NC]
RewriteRule robots\.txt robots-cdn.txt [NC,L]
RewriteCond %{HTTP:Host} !^cdn\.foo\.com$ [NC]
RewriteRule robots\.txt robots-other.txt [NC,L]
You can learn more from examples section
Regards
Andrew
3 posts
• Page 1 of 1
Who is online
Users browsing this forum: No registered users and 0 guests