Rewrite based on http referer and rewritemap

[bobby Profile]

Hi

I want to redirect visitors depending on whether their HTTP Referer matches one of around 2,400 in a text file. I figured the best way to do this would be to use a RewriteMap. It would look like this:

Rewritemap referer txt:referer.txt
If HTTP_Referer matches referer { URL }
Else { URL }

Can anyone help me with the best way to do this? Would it be better to reference an SQL database?

[HeliconAndrew]

Hello,


Please, try using the following:

Code: Select all

RewriteEngine on
RewriteBase /
RewriteMap mapfile txt:mapfile.txt


RewriteCond %{HTTP_REFERER} http://(.*) [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule .* http://${mapfile:%1} [NC,R=301,L]

Mapfile must be located in the same folder with the config file you're using.
the content of a mapfile should look like:

Code: Select all

www.site.com www.otherdomain.com/
site.com/a   www.anything.org/b

Regards
Andrew

[bobby Profile]

Thanks for that Andrew

Is there a way of mapping a large list of URLs to go to one destination instead of doing individual URLs in the mapfile?

Thanks

[HeliconAndrew]

The mapfile is designed in a way that you always need to have something in the second column(after [SPACE]).
So may have

Code: Select all

RewriteEngine on
RewriteBase /
RewriteMap mapfile txt:mapfile.txt


RewriteCond %{HTTP_REFERER} http://(.*) [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule .* http://www.google.com [NC,R=301,L]

and mapfile

Code: Select all

www.site.com   anything
site.com/a     anything

So in this case all of the requests from your list will be redirected to google.com, but you still need to have second column entries in your mapfile.

Regards
Andrew

[bobby Profile]

Thanks Andrew

If I try this:

Code: Select all

RewriteCond %{HTTP_REFERER} http://(.*) [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^/web_page.aspx http://www.test.org.uk [NC,R=301,L]

It seems to ignore the rules entirely?

Also, as I have such a big list, would there be any way of caching this to stop the server getting overloaded?

Thanks!

[HeliconAndrew]

Try:

Code: Select all

RewriteCond %{HTTP_REFERER} http://(.*) [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^web_page\.aspx http://www.test.org.uk [NC,R=301,L]

2400 entries has never been an issue. We've had 20-40,000 of entries.

Regards
Andrew

[bobby Profile]

Thanks for your help Andrew

How can I use a wildcard on the referrer? For example if I have www.test.com in the rewritemap it currently won't redirect if there is a link on www.test.com/test.html

Thanks!

[HeliconAndrew]

Hello,

Actually it already has wildcard. Right now it captures everything and checks the mapfile. The problem could be in the variable itself.
Referer is unreliable as it is set by the clint's browser.

Regards
Andrew

[HeliconAndrew]

Hello,

Actually it already has wildcard. Right now it captures everything and checks the mapfile. The problem could be in the variable itself.
Referer is unreliable as it is set by the clint's browser.

Regards
Andrew

[bobby Profile]

Hi Andrew

Unfortunately we have to use the referer for this task. Is there any way to set a wildcard like this:

RewriteCond ${mapfile:%1(.*)|NOT_FOUND} !NOT_FOUND

So that the URL in the RewriteMap can essentially be represented as www.test.com/(.*)

Thanks

[HeliconAndrew]

Never tried that. My idea would be to tweak the condition, but I don't see why your option wouldn't work.
I'd try:

Code: Select all

RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.])\..* [NC]

However, this will work for www.site.com or site.com, but will fail for subdomain.site.com or site.au.us.

Regards
Andrew

[bobby Profile]

So this won't work for .org.uk, .gov.uk etc? Hmm, any other ideas? I tried it without any luck

Thanks

[HeliconAndrew]

we can create 2 similar rules. one with RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+)\..* [NC]
and another with RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..* [NC]

So that they both will cover most of the domains.

Regards
Andrew

[bobby Profile]

Hi Andrew

I now have the following set of rules:

Code: Select all

RewriteBase /

RewriteMap mapfile txt:mapfile.txt

RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+)\..* [NC]
RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]

In my mapfile I simply have "test.com a"

I am not getting any redirects using the new RewriteConds. Have I implemented this correctly?

Thanks

[HeliconAndrew]

Yep, sorry. I should have put it clearer. You need:

Code: Select all

RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]

RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]

So that you have 2 set of rules for 2 cases.

[bobby Profile]

Hi Andrew

Thanks for that. I'm still not getting a redirect. In the rewritelog I get:

Code: Select all

map lookup FAILED: map=mapfile[txt] key=subdomain.toplevel

Where I would be expecting subdomain.toplevel.org.uk

There are no errors in the error.log

[HeliconAndrew]

Ok, everything is fine here. It's just that you look at the request going through the first rule. It didn't match and it'll go through the second one, where it'll have "subdomain.toplevel.org". So to capture everything in this rule we'll have to add the very last rule to this set:

Code: Select all

 RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]

So in your case you will subsequently checking mapfile three times with 3 values:

subdomain.toplevel
subdomain.toplevel.org
subdomain.toplevel.org.uk

And the last rule should match with the mapfile.

Regards
Andrew

[bobby Profile]

Hi Andrew

Thanks for your help. The rule is being matched but no rewrite is taking place

Code: Select all

Wed, 27-Mar-2013  12:23:52 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20258456/initial] (4) RewriteCond: input='http://subdomain1.toplevel1.org.uk/test.html' pattern='http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..*' => matched

On the next row it says:

Code: Select all

Wed, 27-Mar-2013  12:23:52 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20258456/initial] (5) map lookup FAILED: map=mapfile[txt] key=subdomain1.topleve1.org

FYI my htaccess file now looks like this:

Code: Select all

RewriteEngine on
RewriteBase /

RewriteMap mapfile txt:mapfile.txt

RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]

RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]

RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+\.[^.]+)\..* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]

And my mapfile is simply:

Code: Select all

subdomain1.toplevel1.org.uk www.test.com

[HeliconAndrew]

Seems like I made a small mistake and you didn't correct me. We do not need that last '.' at the end in conditions:

Code: Select all

RewriteEngine on
RewriteBase /

RewriteMap mapfile txt:mapfile.txt

RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+).* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]

RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+).* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]

RewriteCond %{HTTP_REFERER} http://(?:www\.)?([^.]+\.[^.]+\.[^.]+\.[^.]+).* [NC]
RewriteCond ${mapfile:%1|NOT_FOUND} !NOT_FOUND
RewriteRule ^test_page\.aspx http://www.test.org.uk [NC,R=301,L]

Regards
Andrew

[bobby Profile]

Hi Andrew

I get the same error that I did before. Here's the full log (which repeats three times):

Code: Select all

Wed, 27-Mar-2013  15:02:04 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20259936/initial] (4) RewriteCond: input='http://subdomain1.toplevel1.org.uk/test.html' pattern='http://(?:www\.)?([^.]+\.[^.]+).*' => matched

Wed, 27-Mar-2013  15:02:04 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20259936/initial] (5) map lookup FAILED: map=mapfile[txt] key=subdomain1.toplevel1

Wed, 27-Mar-2013  15:02:04 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20259936/initial] (4) RewriteCond: input='NOT_FOUND' pattern='!NOT_FOUND' => not-matched

Wed, 27-Mar-2013  15:02:04 GMT [subdomain.toplevel.org.uk/sid#1863113428][rid#20259936/initial] (3) applying pattern '^test_page\.aspx' to uri 'test_page.aspx'