Everything not in the list of items we’re checking get redirected to the homepage. So a simple typo will get the redirect, but a missing PNG file that’s called by one of our pages won’t send a copy of the homepage to the client telling it it is an image file!

/*
This is a custom 404 handler.

~


So this is handled in Squid.

First, make a simple script.  There’s a possibility another redirector like Squirm might do a better job, but I haven’t played with them.

!/usr/bin/perl
$|=1;
while (<>) {
s@http://www7.getmiro.(com|net|org)/adopt(.*)$@301:https://www7.getmiro.com/adopt$2@;
print;
}

Saved at /etc/squid3/squid_redirector.pl and chown/chmod so the user “proxy” that squid runs under can run it.  Your path, of course, may vary.

The key part for what we need is that we pre-pend “301″ before https:  in the rewrite.  When this is returned to the user’s browser it redirects them to the secure page.  This script also takes anything at com, net, or org and forces them to a tld of .com as well.

It’s easy to test this perl script.  Simply type ./squid_redirector.pl which launches it interactively.

# ./squid_redirector.pl
http://www7.getmiro.com/foo

http://www7.getmiro.com/foo

http://www7.getmiro.com/adopt/test
301:https://www7.getmiro.com/adopt/test
http://www7.getmiro.net/adopt/matt/is/an/evil/genius
301:https://www7.getmiro.com/adopt/matt/is/an/evil/genius

Next, tell Squid to use it.  We need to enable these lines in the squid.conf file:

url_rewrite_program /etc/squid3/squid_redirector.pl
url_rewrite_children 10
url_rewrite_host_header off
url_rewrite_bypass on

The first line tells Squid what to use to rewrite URLs, the second tells it to spawn 10 instances on startup.  I’m not sure, in the end, if host_header needs to be off.  url_rewrite_bypass on allows Squid to skip the re-writing step if all the redirectors are busy.  That’s a decision knowing our security risks, users, and needs — and I’m going with more reliability over absolute security.  We’ll should see skips showing up in the logs and adjust settings from there if necessary.

Restart Squid, give it a test.  Famous last words — it should work now.

References:

http://wiki.squid-cache.org/Features/Redirectors

http://brainextender.blogspot.com/2009/01/simple-squid-redirector-perl-script.html

But the standard format of a Lighttpd virtual host entry doesn’t recognize alternate ports appended after the tld.  Not a big deal, this does the trick:

$HTTP["host"] =~ “(^|\.)getmiro\.(com|net|org)($|:8080$)” {

Translated:
(^|\.) Any hostname
getmiro\ Going to the gemtiro domain
.(com|net|org) with a top level domain of com, net, or org
($|:8080$){ and ending with the tld or :8080 will be processed by the rules that follow.

To avoid that, it’s a simple wildcard entry like this in the appropriate named database:

*.mirocommunity.com.    IN      CNAME   mirocommunity.com.

Which directs everything to our server.

Now our server hosts multiple sites via host entries, so we can’t use a simple negation like this:

$HTTP["host"] !~ “^(www|medfield)\.mirocommunity\.(com|net|org)($|:8080$)” {
url.redirect = (
“^(.*)$” => “http://www.mirocommunity.com$1″,
)
}

Note the negation by using !~ instead of =~.  That would work if all we had was mirocommunity sites to host, but when hitting another site on the server like www7.getmiro.com it would read it as not being www or medfield dot mirocommunity, and thus drop you to www.mirocommunity.com.  For the curious, the 8080 part of the url parsing is a bypass of the Squid proxies on ports 80 and 443.

Anything that doesn’t match a virtual host or alias on our server gets dropped by default to /var/www.

There lies the simple solution — put an index.php file there that does the redirect work:

<?php
// Hostnames that aren’t matched in Lighttpd get dropped here
// by default.
// This script removes the hostname(s) and drops them to
// www.[domain].[tld]
// 29 May 2009 MRK
$split_host = split(“\.”, $_SERVER[HTTP_HOST]);
$domain = count($split_host) – 2;
$tld = count($split_host) – 1;
$new_host = “http://www.$split_host[$domain].$split_host[$tld]“;
// echo “$new_host”;
header(“Location: $new_host”);
exit;
?>

The split command splits the $_SERVER[HTTP_HOST] variable at each period, and put it’s contents less the periods into an array called $split_host.

The count($split_host) determines how many members we have in the $split_host array.  We know we always want the last (the top level domain — .com, etc) and second to last (the domain — mirocommunity, etc).  Since arrays start at 0, we simply count -1 for the tld and -2 for the domain.

By adding the count logic, we can handle domains like brooklyn.newyork.mirocommunity.com which have more then one hostname before the domain and tld.

$new_host then forms the URL we want to catch wildcard hostnames that haven’t been configured yet.  It’s simply the www.domain.tld form.  That’s fed to a http header which causes the user’s browser to redirect to the default website we want.

So as of today, while newyork.mirocommunity.com and brooklyn.newyork.mirocommunity.com have no virtual hosts, you do arrive successfully at www.mirocommunity.com.

Our developers can activate those hostnames simply by adding an entry in the appropriate lighttpd conf file and reload lighttpd — no need to contact the sysadmin to go make an entry in our DNS system for each new city added.

To answer a couple questions off the top, you should also read my post on how to configure http –> https redirects at the Squid level since the web server won’t be able to handle that in this configuration, and this post documents a little bit of magic that needs to be done to support 8080 with virtual hosts.

In configuring our new servers, the choice of Squid was pretty easy — it can handle SSL traffic, Varnish can’t by itself.  We already use Squid to do ssl traffic on some of our physical servers being replaced, so I’d like to continue using that feature.  In a future post, we’ll talk about configuring Squid to use a Universal Certificate that can handle multiple domains on one IP (it looks doable in theory, but I haven’t purchased that yet).

Normally installation is a simple

apt-get install squid

to install Squid.  However, Ubuntu doesn’t package OpenSSL with Squid and for license reasons has no intention of doing so.  So you’re better off following these directions and modifying a package to include ssl support, then installing that.

Modify /etc/squid/squid.conf

This is the port 80 traffic.  Note — we actually had a large number of “acl valid_dst dstdomain,” which block attempts to use Squid as a pass-thru proxy at the proxy level instead of having the webserver reject the traffic.

http_port 80 accel vhost

cache_peer 127.0.0.1 parent 8080 0 no-query originserver login=PASS

logformat combined %>a %ui %un [%{%d/%b/%Y:%H:%M:%S +0500}tl] “%rm %ru HTTP/%rv” %Hs %h” “%{User-Agent}>h” %Ss:%Shaccess_log /var/log/squid/access.log combined

acl SSL_ports port 8080

acl Safe_ports port 8080

acl valid_dst dstdomain .somedomain.com

http_access allow valid_dst

Copy squid.conf to squid_ssl.conf, comment out http_port and make the following changes:

https_port 443 accel vhost cert=/(cert location) key=/(key location)
cache_log /var/log/squid3/cache_ssl.log
cache_store_log /var/log/squid3/store_ssl.log

We have seperate cache and store logs for troubleshooting, but both configurations use access.log to record traffic.  This simplifies using AWStats to analyze the logs; if we run into performance problems in the future we may need a tool like logmerge.pl to consolidate seperate access logs.  While I can think of a few things that could go wrong, I don’t know they will go wrong till we try, so let’s see if the simple way works first.

Now, let’s configure and initialize a seperate spool for ssl traffic:

mkdir /var/spool/squid3_ssl
chown -R proxy:proxy /var/spool/squid3_ssl/
squid3 -z -f /etc/squid3/squid_ssl.conf

Copy /etc/init.d/squid3 to /etc/init.d/squid3_ssl and make the following changes:

NAME=squid3_ssl
SQUID_ARGS=”-D -YC -f /etc/squid3/squid_ssl.conf”
CONFIG=/etc/squid3/squid_ssl.conf
$DAEMON -z -f $CONFIG

And do a ln -s /etc/init.d/squid3_ssl /etc/rc2.d/S30squid3_ssl to make it start automatically.

This accomplishes two things — it improves the performance for our visitors since Amazon has faster performance and reliability then we can afford on our own servers, and it does so at a lower cost.

In this post we’ll look at how much bandwidth and files/redirect we use without S3, then with various combination of local and redirected files, up to code as optimized as I have been able to make it — fully optimized we our servers only transfer 6.7% of the bytes that the “unoptimized” site would.  Optimizing this single popular page to use S3 efficiently saves PCF about $1,000 a year in hosting costs.

1)
Let’s look at a redacted Squid log after the February, 2009 redesign of www.getmiro.com when not using S3 at all:

"GET http://www.getmiro.com/ HTTP/1.1" 200 21781 "-" "Mozilla/5.0
 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6"
"GET http://www.getmiro.com//css/nav.css HTTP/1.1" 200 5004 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com//css/styles.css HTTP/1.1" 200 21515 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com/css/index.css HTTP/1.1" 200 7709 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com//i/blue_bg.png HTTP/1.1" 200 1198 "http://www.getmiro.com//css/styles.css" "Mozilla/5.0
"GET http://www.getmiro.com//i/nav_back.gif HTTP/1.1" 200 530 "http://www.getmiro.com//css/nav.css" "Mozilla/5.0
[blah blah blah...]

That’s 37 files, for a total of 338,359 Bytes.

2)
Now let’s look if we load CSS from our server, but use Apache to re-write images and js to the S3 service:

"GET http://www.getmiro.com/ HTTP/1.1" 200 21781 "http://www.getmiro.com/" "Mozilla/5.0
 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6"
"GET http://www.getmiro.com//css/nav.css HTTP/1.1" 200 5004 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com//css/styles.css HTTP/1.1" 200 21515 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com/css/index.css HTTP/1.1" 200 7709 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com//i/blue_bg.png HTTP/1.1" 302 688 "http://www.getmiro.com//css/styles.css" "Mozilla/5.0
"GET http://www.getmiro.com//i/nav_back.gif HTTP/1.1" 302 690 "http://www.getmiro.com//css/nav.css" "Mozilla/5.0
[blah blah blah...]

Now it’s four files, plus 33 redirects — and only 78,705 bytes.

3)
Now let’s use Apache to redirect the CSS to be pulled from Amazon S3.

"GET http://www.getmiro.com/ HTTP/1.1" 200 21781 "-" "Mozilla/4.0
 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
"GET http://www.getmiro.com//css/nav.css HTTP/1.1" 302 684 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com//css/styles.css HTTP/1.1" 302 690 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com/css/index.css HTTP/1.1" 302 688 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com//i/blue_bg.png HTTP/1.1" 302 687 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com//i/nav_back.gif HTTP/1.1" 302 689 "http://www.getmiro.com/" "Mozilla/4.0
[blah blah blah...]

Still four files and 33 redirects, but down to 46,506 bytes.

So far it’s all been pretty standard stuff in Apache using mod_rewrite redirects.  Apache sees a CSS sheet being called, it redirects it to S3.

RewriteRule ^/css/(.*) http://s3.getmiro.3.0.com.s3.amazonaws.com/css/$1

And then a css sheet may have a line like this:

background: url(../i/screen_dropshadow.png) -20px -36px no-repeat;

Now an observant user may note in the logs above that I switched from using Firefox to IE.  Why?  The browsers interpret the CSS differently.

Firefox interprets “../i/” relative to where the CSS style sheet is LOADED from — in our case  http://s3.getmiro.3.0.com.s3.amazonaws.com/css/.
Internet Explorer interprets “../i/” relative to where the CSS style sheet is CALLED from — in our case http://www.getmiro.com/css/.

Those familiar with unix notation know that “../i/” from “getmiro.com/css/” gets you to “getmiro.com/i/”.

4)
Now we get fancy.

In implementing S3, we have a bash script which handles the synchronization between our servers and S3.  So in that script, let’s intercept the CSS sheets, do a simple SED, and upload the modified files to a special location:

# getmiro css
# This substitutes ../i with http://s3.getmiro.3.0.com.s3.amazonaws.com/ in the getmiro css code
# and uploads them to a special directory in amazon.  This is in turn re-written by Apache to point there.
# Having the full url hard coded saves tens of thousands of redirects and gigs of bandwidth.
# It's also more efficient then "php-ifying" css to do the url substitution.

  # First, copy they css to a working directory:
    cp /data/getmiro/css/*.css /scripts/getmiro_css

  # It's safer to just modify files we know about, rather then automate finding and modifying without foreknowledge:
    sed -i 's/\.\.\/i/http:\/\/s3.getmiro.3.0.com.s3.amazonaws.com\/i/g' /scripts/getmiro_css/download-features.css
    sed -i 's/\.\.\/i/http:\/\/s3.getmiro.3.0.com.s3.amazonaws.com\/i/g' /scripts/getmiro_css/index.css
    sed -i 's/\.\.\/i/http:\/\/s3.getmiro.3.0.com.s3.amazonaws.com\/i/g' /scripts/getmiro_css/nav.css
    sed -i 's/\.\.\/i/http:\/\/s3.getmiro.3.0.com.s3.amazonaws.com\/i/g' /scripts/getmiro_css/styles.css

  # And let's upload them:
    /usr/local/s3sync/s3sync.rb -r -p -v /scripts/getmiro_css/ s3.getmiro.3.0.com:css/s3_coded/

In the background so the web developers don’t have to worry about modifying the CSS sheets to include the hard link, transforming lines like:

background: url(../i/screen_dropshadow.png) -20px -36px no-repeat;

into

background: url(s3.getmiro.3.0.com.s3.amazonaws.com/i/screen_dropshadow.png) -20px -36px no-repeat;

It’s necessary to use the pattern \.\./i/ in sed in case a developer does hard code the amazon link.  The \. means literally a period; regexes like this otherwise use a . as a wildcard to match one character, and just ../i/ would match any two characters before /i/.

In Apache, we change the redirect to this:

RewriteRule ^/css/(.*) http://s3.getmiro.3.0.com.s3.amazonaws.com/css/S3_coded/$1

With this change implemented:

"GET http://www.getmiro.com/ HTTP/1.1" 200 21781 "-" "Mozilla/4.0
 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
"GET http://www.getmiro.com//css/styles.css HTTP/1.1" 302 708 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com//css/nav.css HTTP/1.1" 302 702 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com/css/index.css HTTP/1.1" 302 706 "http://www.getmiro.com/" "Mozilla/4.0

Much better!  One file, three redirects for 23,897 bytes.  That one change represents nearly a 50% reduction in bandwidth usage on our physical servers from just the example above, and only about 1/3rd the bandwidth if we used CSS sheets being served locally even if they had the hard links to S3 on them.

5)
Finally one more tweak.

The default Apache redirect includes HTML code saying where a file has moved.  But this isn’t necessary — a web browser just needs the correct headers to tell it where to go.

So replacing the redirect to S3 we used before, we use this:

RewriteRule ^/css/(.*) /custom_messages/css_rewrite.php

This is css_rewrite.php:

<?php
/*
This rewrites css just using headers.  This saves about 300bytes per
redirect -- which saves a heck of a lot of bandwidth over time when we're doing 3
css rewrites for every page view...works out to 30+ MB / day!

Invoke by:
RewriteRule ^/css/(.*) /custom_messages/css_rewrite.php
*/

$new_server = "http://s3.getmiro.3.0.com.s3.amazonaws.com/css/s3_coded/";
$new_url = preg_replace('/^.*\//', $new_server, $_SERVER[REQUEST_URI]);

echo Header( "HTTP/1.1 301 Moved Permanently" );
echo Header( "Location: $new_url" );
?>

This produces a very minimal redirect — under 400 bytes rather then over 700 bytes.

I haven’t done a complete analysis to know if this significantly slower then a native Apache redirect; initial review shows it is not slower for any given page load.  This step would need a very, very busy site however to make a meaningful performance or cost impact.  It’s something I’m noting though, because there could be other situations this type of optimization could be useful.  Most importantly the variables Apache (or IIS) can pass to other programs like PHP.  See this link for a list of them.

"GET http://www.getmiro.com/ HTTP/1.1" 200 21781 "-" "Mozilla/4.0
 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
"GET http://www.getmiro.com//css/styles.css HTTP/1.1" 301 397 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com//css/nav.css HTTP/1.1" 301 394 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com/css/index.css HTTP/1.1" 301 396 "http://www.getmiro.com/" "Mozilla/4.0

Now just one file, three redirects and 22,968.

Bottom line?

Let’s take a typical day when the getmiro.com homepage is called 20,000 times.

Scenario    Size     Total Daily      Estimated
                       Bandwidth     Daily Cost**
1           338,359        6.5GB          $4.73
2            78,705        1.5GB           1.09
3            46,506        887MB*           .65
4            23,897        455MB            .33
5            22,968        438MB            .32
* Unadjusted for Firefox's interpretation of CSS paths
** This estimate is based on purchasing enough fixed bandwidth (Mbps)
to cover our peak daily usage.  Our communication costs for our
physical servers is approximately times as much as Amazon S3 based on
actual transfers.

So without S3 or any optimization, we’d be looking at a monthly cost around $142.00.

With S3 and with all our optimization, we’re looking at a monthly cost around $54.00.

For a small non-profit, that’s a nice savings over time.

Information Needed:
Country Name (2 letter code) [AU]:
State or Province Name (full name) [Some-State]:
Locality Name (e.g., city) [ ]:
Organization Name (e.g., company) [Internet Widgets Pty Ltd]:
Organizational Unit Name (e.g., section) [ ]:
Common Name (e.g., YOUR name) [ ]: (See my email...)
Email Address [ ]:

Generate a CSR from an existing private key:

 openssl req -new -key private-key.pem -out /home/rfagundo/csr_2007.pem

==========================================
Some handy commands:

View the key:
openssl rsa -noout -text -in name.key

View the CSR:
openssl req -noout -text -in name.csr

View the Certificate:
openssl x509 -noout -text -in name.crt

Modulus (+ Exponent) should match between the key and the others…otherwise you get a key mismatch

==========================================
Squid Reverse Proxy & SSL:

Squid can’t handle seperate chainfiles.  But it’s a pretty easy fix.

Use this sample for an Apache conf file for a site being moved to Squid:

SSLEngine on
SSLCertificateFile /etc/httpd/ssl/mysite/www.mysite.com.crt
SSLCertificateKeyFile /etc/httpd/ssl/mysite/mysite.key
SSLCertificateChainFile /etc/httpd/ssl/mysite/sf_intermediate.crt

This process certainly can be modified / simplified as needed:
Copy the www.mysite.com.crt to squid_www.mysite.com.crt
Copy the text from sf_intermediate.crt and paste it to the bottom of squid_www.mysite.com.crt.

Here’s the Squid SSL line that uses those:

https_port 8081 cert=/etc/httpd/ssl/mysite/squid_www.mysite.com.crt key=/etc/httpd/ssl/mysite/mysite.key vhost defaultsite=www.mysite.com

BTW, in the above sample…
public ip:443 = Apache running SSL
public ip:8081 = Squid running SSL, which then connects to 127.0.0.1:8080 on Apache *not* running SSL.

In production, you take public ip:443 off apache, and have Squid listen on public ip:443 instead.

Download the rpm.bin from Sun.  When you run it, it will install the RPM too.  http://www.java.com/en/download/manual.jsp

Now, update the java and javac called by default, via update-alternatives:

The “1500″ is the priority.  Higher number = precedence (on the machine I was installling on, the existing java was priority 1420)

And you may want to use “find / -name java”, etc to find the actual locations on your system for linking in the new versions.

This seems like a good reference, too:  http://www.gagme.com/greg/linux/fc6-tips.php

# To display (and verify) settings before and after:
/usr/sbin/update-alternatives –display java

# Java itself:
/usr/sbin/update-alternatives –install \
/usr/bin/java java /usr/java/jdk1.5.0_12/jre/bin/java 1500 \
–slave /usr/bin/keytool keytool /usr/java/jdk1.5.0_12/jre/bin/keytool \
–slave /usr/bin/rmiregistry rmiregistry /usr/java/jdk1.5.0_12/jre/bin/rmiregistry

# To display (and verify) settings before and after:
  /usr/sbin/update-alternatives –display javac

# Javac:
/usr/sbin/update-alternatives –install \
/usr/bin/javac javac /usr/java/jdk1.5.0_12/bin/javac 1500 \
–slave /usr/bin/javadoc javadoc /usr/java/jdk1.5.0_12/bin/javadoc \
–slave /usr/bin/javah javah /usr/java/jdk1.5.0_12/bin/javah \
–slave /usr/bin/jar jar /usr/java/jdk1.5.0_12/bin/jar \
–slave /usr/bin/jarsigner jarsigner /usr/java/jdk1.5.0_12/bin/jarsigner \
–slave /usr/bin/appletviewer appletviewer /usr/java/jdk1.5.0_12/bin/appletviewer \
–slave /usr/bin/rmic rmic /usr/java/jdk1.5.0_12/bin/rmic
  # These *may* be needed…doesn’t seem to be used on the system I’m working on (i.e. no java_jdk actually exist outside of /etc/alternatives)
  # –slave /usr/lib/jvm/java java_sdk $sun/jdk \
  #–slave /usr/lib/jvm-exports/java java_sdk_exports $sun/jdk/jre/lib