Rewrite based on http referer and rewritemap
26 posts
• Page 1 of 2 • 1, 2
Rewrite based on http referer and rewritemap
Hi
I want to redirect visitors depending on whether their HTTP Referer matches one of around 2,400 in a text file. I figured the best way to do this would be to use a RewriteMap. It would look like this:
Rewritemap referer txt:referer.txt
If HTTP_Referer matches referer { URL }
Else { URL }
Can anyone help me with the best way to do this? Would it be better to reference an SQL database?
I want to redirect visitors depending on whether their HTTP Referer matches one of around 2,400 in a text file. I figured the best way to do this would be to use a RewriteMap. It would look like this:
Rewritemap referer txt:referer.txt
If HTTP_Referer matches referer { URL }
Else { URL }
Can anyone help me with the best way to do this? Would it be better to reference an SQL database?
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Rewrite based on http referer and rewritemap
Hello,
Please, try using the following:
Mapfile must be located in the same folder with the config file you're using.
the content of a mapfile should look like:
Regards
Andrew
Please, try using the following:
- Code: Select all
RewriteEngine on
RewriteBase /
RewriteMap mapfile txt:mapfile.txt
RewriteCond %{HTTP_REFERER} http://(.*) [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule .* http://${mapfile:%1} [NC,R=301,L]
Mapfile must be located in the same folder with the config file you're using.
the content of a mapfile should look like:
- Code: Select all
www.site.com www.otherdomain.com/
site.com/a www.anything.org/b
Regards
Andrew
Re: Rewrite based on http referer and rewritemap
Thanks for that Andrew
Is there a way of mapping a large list of URLs to go to one destination instead of doing individual URLs in the mapfile?
Thanks
Is there a way of mapping a large list of URLs to go to one destination instead of doing individual URLs in the mapfile?
Thanks
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Rewrite based on http referer and rewritemap
The mapfile is designed in a way that you always need to have something in the second column(after [SPACE]).
So may have
and mapfile
So in this case all of the requests from your list will be redirected to google.com, but you still need to have second column entries in your mapfile.
Regards
Andrew
So may have
- Code: Select all
RewriteEngine on
RewriteBase /
RewriteMap mapfile txt:mapfile.txt
RewriteCond %{HTTP_REFERER} http://(.*) [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule .* http://www.google.com [NC,R=301,L]
and mapfile
- Code: Select all
www.site.com anything
site.com/a anything
So in this case all of the requests from your list will be redirected to google.com, but you still need to have second column entries in your mapfile.
Regards
Andrew
Re: Rewrite based on http referer and rewritemap
Thanks Andrew
If I try this:
It seems to ignore the rules entirely?
Also, as I have such a big list, would there be any way of caching this to stop the server getting overloaded?
Thanks!
If I try this:
- Code: Select all
RewriteCond %{HTTP_REFERER} http://(.*) [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^/web_page.aspx http://www.test.org.uk [NC,R=301,L]
It seems to ignore the rules entirely?
Also, as I have such a big list, would there be any way of caching this to stop the server getting overloaded?
Thanks!
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Rewrite based on http referer and rewritemap
Try:
2400 entries has never been an issue. We've had 20-40,000 of entries.
Regards
Andrew
- Code: Select all
RewriteCond %{HTTP_REFERER} http://(.*) [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^web_page\.aspx http://www.test.org.uk [NC,R=301,L]
2400 entries has never been an issue. We've had 20-40,000 of entries.
Regards
Andrew
Re: Rewrite based on http referer and rewritemap
Thanks for your help Andrew
How can I use a wildcard on the referrer? For example if I have www.test.com in the rewritemap it currently won't redirect if there is a link on www.test.com/test.html
Thanks!
How can I use a wildcard on the referrer? For example if I have www.test.com in the rewritemap it currently won't redirect if there is a link on www.test.com/test.html
Thanks!
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Rewrite based on http referer and rewritemap
Hello,
Actually it already has wildcard. Right now it captures everything and checks the mapfile. The problem could be in the variable itself.
Referer is unreliable as it is set by the clint's browser.
Regards
Andrew
Actually it already has wildcard. Right now it captures everything and checks the mapfile. The problem could be in the variable itself.
Referer is unreliable as it is set by the clint's browser.
Regards
Andrew
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Rewrite based on http referer and rewritemap
Hello,
Actually it already has wildcard. Right now it captures everything and checks the mapfile. The problem could be in the variable itself.
Referer is unreliable as it is set by the clint's browser.
Regards
Andrew
Actually it already has wildcard. Right now it captures everything and checks the mapfile. The problem could be in the variable itself.
Referer is unreliable as it is set by the clint's browser.
Regards
Andrew
Re: Rewrite based on http referer and rewritemap
Hi Andrew
Unfortunately we have to use the referer for this task. Is there any way to set a wildcard like this:
RewriteCond ${mapfile:%1(.*)|NOT_FOUND} !NOT_FOUND
So that the URL in the RewriteMap can essentially be represented as www.test.com/(.*)
Thanks
Unfortunately we have to use the referer for this task. Is there any way to set a wildcard like this:
RewriteCond ${mapfile:%1(.*)|NOT_FOUND} !NOT_FOUND
So that the URL in the RewriteMap can essentially be represented as www.test.com/(.*)
Thanks
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Rewrite based on http referer and rewritemap
Never tried that. My idea would be to tweak the condition, but I don't see why your option wouldn't work.
I'd try:
However, this will work for www.site.com or site.com, but will fail for subdomain.site.com or site.au.us.
Regards
Andrew
I'd try:
- Code: Select all
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.])\..* [NC]
However, this will work for www.site.com or site.com, but will fail for subdomain.site.com or site.au.us.
Regards
Andrew
Re: Rewrite based on http referer and rewritemap
So this won't work for .org.uk, .gov.uk etc? Hmm, any other ideas? I tried it without any luck
Thanks
Thanks
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Rewrite based on http referer and rewritemap
we can create 2 similar rules. one with RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+)\..* [NC]
and another with RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..* [NC]
So that they both will cover most of the domains.
Regards
Andrew
and another with RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..* [NC]
So that they both will cover most of the domains.
Regards
Andrew
Re: Rewrite based on http referer and rewritemap
Hi Andrew
I now have the following set of rules:
In my mapfile I simply have "test.com a"
I am not getting any redirects using the new RewriteConds. Have I implemented this correctly?
Thanks
I now have the following set of rules:
- Code: Select all
RewriteBase /
RewriteMap mapfile txt:mapfile.txt
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+)\..* [NC]
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]
In my mapfile I simply have "test.com a"
I am not getting any redirects using the new RewriteConds. Have I implemented this correctly?
Thanks
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Rewrite based on http referer and rewritemap
Yep, sorry. I should have put it clearer. You need:
So that you have 2 set of rules for 2 cases.
- Code: Select all
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]
So that you have 2 set of rules for 2 cases.
Re: Rewrite based on http referer and rewritemap
Hi Andrew
Thanks for that. I'm still not getting a redirect. In the rewritelog I get:
Where I would be expecting subdomain.toplevel.org.uk
There are no errors in the error.log
Thanks for that. I'm still not getting a redirect. In the rewritelog I get:
- Code: Select all
map lookup FAILED: map=mapfile[txt] key=subdomain.toplevel
Where I would be expecting subdomain.toplevel.org.uk
There are no errors in the error.log
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Rewrite based on http referer and rewritemap
Ok, everything is fine here. It's just that you look at the request going through the first rule. It didn't match and it'll go through the second one, where it'll have "subdomain.toplevel.org". So to capture everything in this rule we'll have to add the very last rule to this set:
So in your case you will subsequently checking mapfile three times with 3 values:
And the last rule should match with the mapfile.
Regards
Andrew
- Code: Select all
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]
So in your case you will subsequently checking mapfile three times with 3 values:
subdomain.toplevel
subdomain.toplevel.org
subdomain.toplevel.org.uk
And the last rule should match with the mapfile.
Regards
Andrew
Re: Rewrite based on http referer and rewritemap
Hi Andrew
Thanks for your help. The rule is being matched but no rewrite is taking place
On the next row it says:
FYI my htaccess file now looks like this:
And my mapfile is simply:
Thanks for your help. The rule is being matched but no rewrite is taking place
- Code: Select all
Wed, 27-Mar-2013 12:23:52 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20258456/initial] (4) RewriteCond: input='http://subdomain1.toplevel1.org.uk/test.html' pattern='http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..*' => matched
On the next row it says:
- Code: Select all
Wed, 27-Mar-2013 12:23:52 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20258456/initial] (5) map lookup FAILED: map=mapfile[txt] key=subdomain1.topleve1.org
FYI my htaccess file now looks like this:
- Code: Select all
RewriteEngine on
RewriteBase /
RewriteMap mapfile txt:mapfile.txt
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]
And my mapfile is simply:
- Code: Select all
subdomain1.toplevel1.org.uk www.test.com
- HeliconAndrew
- Posts: 1264
- Joined: 07 Mar 2012, 10:16
Re: Rewrite based on http referer and rewritemap
Seems like I made a small mistake and you didn't correct me. We do not need that last '.' at the end in conditions:
Regards
Andrew
- Code: Select all
RewriteEngine on
RewriteBase /
RewriteMap mapfile txt:mapfile.txt
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+).* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+).* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+\.[^.]+).* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]
Regards
Andrew
Re: Rewrite based on http referer and rewritemap
Hi Andrew
I get the same error that I did before. Here's the full log (which repeats three times):
I get the same error that I did before. Here's the full log (which repeats three times):
- Code: Select all
Wed, 27-Mar-2013 15:02:04 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20259936/initial] (4) RewriteCond: input='http://subdomain1.toplevel1.org.uk/test.html' pattern='http://(?:www\.)?([^.]+\.[^.]+).*' => matched
Wed, 27-Mar-2013 15:02:04 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20259936/initial] (5) map lookup FAILED: map=mapfile[txt] key=subdomain1.toplevel1
Wed, 27-Mar-2013 15:02:04 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20259936/initial] (4) RewriteCond: input='NOT_FOUND' pattern='!NOT_FOUND' => not-matched
Wed, 27-Mar-2013 15:02:04 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20259936/initial] (3) applying pattern '^test_page\.aspx' to uri 'test_page.aspx'
26 posts
• Page 1 of 2 • 1, 2
Who is online
Users browsing this forum: No registered users and 10 guests