Everything not in the list of items we’re checking get redirected to the homepage. So a simple typo will get the redirect, but a missing PNG file that’s called by one of our pages won’t send a copy of the homepage to the client telling it it is an image file!

/*
This is a custom 404 handler.

~


Linux:
Rootkit Hunter
Chkrootkit

Windows:
RootkitRevealer

So this is handled in Squid.

First, make a simple script.  There’s a possibility another redirector like Squirm might do a better job, but I haven’t played with them.

!/usr/bin/perl
$|=1;
while (<>) {
s@http://www7.getmiro.(com|net|org)/adopt(.*)$@301:https://www7.getmiro.com/adopt$2@;
print;
}

Saved at /etc/squid3/squid_redirector.pl and chown/chmod so the user “proxy” that squid runs under can run it.  Your path, of course, may vary.

The key part for what we need is that we pre-pend “301″ before https:  in the rewrite.  When this is returned to the user’s browser it redirects them to the secure page.  This script also takes anything at com, net, or org and forces them to a tld of .com as well.

It’s easy to test this perl script.  Simply type ./squid_redirector.pl which launches it interactively.

# ./squid_redirector.pl
http://www7.getmiro.com/foo

http://www7.getmiro.com/foo

http://www7.getmiro.com/adopt/test
301:https://www7.getmiro.com/adopt/test
http://www7.getmiro.net/adopt/matt/is/an/evil/genius
301:https://www7.getmiro.com/adopt/matt/is/an/evil/genius

Next, tell Squid to use it.  We need to enable these lines in the squid.conf file:

url_rewrite_program /etc/squid3/squid_redirector.pl
url_rewrite_children 10
url_rewrite_host_header off
url_rewrite_bypass on

The first line tells Squid what to use to rewrite URLs, the second tells it to spawn 10 instances on startup.  I’m not sure, in the end, if host_header needs to be off.  url_rewrite_bypass on allows Squid to skip the re-writing step if all the redirectors are busy.  That’s a decision knowing our security risks, users, and needs — and I’m going with more reliability over absolute security.  We’ll should see skips showing up in the logs and adjust settings from there if necessary.

Restart Squid, give it a test.  Famous last words — it should work now.

References:

http://wiki.squid-cache.org/Features/Redirectors

http://brainextender.blogspot.com/2009/01/simple-squid-redirector-perl-script.html

But the standard format of a Lighttpd virtual host entry doesn’t recognize alternate ports appended after the tld.  Not a big deal, this does the trick:

$HTTP["host"] =~ “(^|\.)getmiro\.(com|net|org)($|:8080$)” {

Translated:
(^|\.) Any hostname
getmiro\ Going to the gemtiro domain
.(com|net|org) with a top level domain of com, net, or org
($|:8080$){ and ending with the tld or :8080 will be processed by the rules that follow.

To avoid that, it’s a simple wildcard entry like this in the appropriate named database:

*.mirocommunity.com.    IN      CNAME   mirocommunity.com.

Which directs everything to our server.

Now our server hosts multiple sites via host entries, so we can’t use a simple negation like this:

$HTTP["host"] !~ “^(www|medfield)\.mirocommunity\.(com|net|org)($|:8080$)” {
url.redirect = (
“^(.*)$” => “http://www.mirocommunity.com$1″,
)
}

Note the negation by using !~ instead of =~.  That would work if all we had was mirocommunity sites to host, but when hitting another site on the server like www7.getmiro.com it would read it as not being www or medfield dot mirocommunity, and thus drop you to www.mirocommunity.com.  For the curious, the 8080 part of the url parsing is a bypass of the Squid proxies on ports 80 and 443.

Anything that doesn’t match a virtual host or alias on our server gets dropped by default to /var/www.

There lies the simple solution — put an index.php file there that does the redirect work:

<?php
// Hostnames that aren’t matched in Lighttpd get dropped here
// by default.
// This script removes the hostname(s) and drops them to
// www.[domain].[tld]
// 29 May 2009 MRK
$split_host = split(“\.”, $_SERVER[HTTP_HOST]);
$domain = count($split_host) – 2;
$tld = count($split_host) – 1;
$new_host = “http://www.$split_host[$domain].$split_host[$tld]“;
// echo “$new_host”;
header(“Location: $new_host”);
exit;
?>

The split command splits the $_SERVER[HTTP_HOST] variable at each period, and put it’s contents less the periods into an array called $split_host.

The count($split_host) determines how many members we have in the $split_host array.  We know we always want the last (the top level domain — .com, etc) and second to last (the domain — mirocommunity, etc).  Since arrays start at 0, we simply count -1 for the tld and -2 for the domain.

By adding the count logic, we can handle domains like brooklyn.newyork.mirocommunity.com which have more then one hostname before the domain and tld.

$new_host then forms the URL we want to catch wildcard hostnames that haven’t been configured yet.  It’s simply the www.domain.tld form.  That’s fed to a http header which causes the user’s browser to redirect to the default website we want.

So as of today, while newyork.mirocommunity.com and brooklyn.newyork.mirocommunity.com have no virtual hosts, you do arrive successfully at www.mirocommunity.com.

Our developers can activate those hostnames simply by adding an entry in the appropriate lighttpd conf file and reload lighttpd — no need to contact the sysadmin to go make an entry in our DNS system for each new city added.

This accomplishes two things — it improves the performance for our visitors since Amazon has faster performance and reliability then we can afford on our own servers, and it does so at a lower cost.

In this post we’ll look at how much bandwidth and files/redirect we use without S3, then with various combination of local and redirected files, up to code as optimized as I have been able to make it — fully optimized we our servers only transfer 6.7% of the bytes that the “unoptimized” site would.  Optimizing this single popular page to use S3 efficiently saves PCF about $1,000 a year in hosting costs.

1)
Let’s look at a redacted Squid log after the February, 2009 redesign of www.getmiro.com when not using S3 at all:

"GET http://www.getmiro.com/ HTTP/1.1" 200 21781 "-" "Mozilla/5.0
 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6"
"GET http://www.getmiro.com//css/nav.css HTTP/1.1" 200 5004 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com//css/styles.css HTTP/1.1" 200 21515 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com/css/index.css HTTP/1.1" 200 7709 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com//i/blue_bg.png HTTP/1.1" 200 1198 "http://www.getmiro.com//css/styles.css" "Mozilla/5.0
"GET http://www.getmiro.com//i/nav_back.gif HTTP/1.1" 200 530 "http://www.getmiro.com//css/nav.css" "Mozilla/5.0
[blah blah blah...]

That’s 37 files, for a total of 338,359 Bytes.

2)
Now let’s look if we load CSS from our server, but use Apache to re-write images and js to the S3 service:

"GET http://www.getmiro.com/ HTTP/1.1" 200 21781 "http://www.getmiro.com/" "Mozilla/5.0
 (Windows; U; Windows NT 5.1; en-US; rv:1.9.0.6) Gecko/2009011913 Firefox/3.0.6"
"GET http://www.getmiro.com//css/nav.css HTTP/1.1" 200 5004 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com//css/styles.css HTTP/1.1" 200 21515 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com/css/index.css HTTP/1.1" 200 7709 "http://www.getmiro.com/" "Mozilla/5.0
"GET http://www.getmiro.com//i/blue_bg.png HTTP/1.1" 302 688 "http://www.getmiro.com//css/styles.css" "Mozilla/5.0
"GET http://www.getmiro.com//i/nav_back.gif HTTP/1.1" 302 690 "http://www.getmiro.com//css/nav.css" "Mozilla/5.0
[blah blah blah...]

Now it’s four files, plus 33 redirects — and only 78,705 bytes.

3)
Now let’s use Apache to redirect the CSS to be pulled from Amazon S3.

"GET http://www.getmiro.com/ HTTP/1.1" 200 21781 "-" "Mozilla/4.0
 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
"GET http://www.getmiro.com//css/nav.css HTTP/1.1" 302 684 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com//css/styles.css HTTP/1.1" 302 690 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com/css/index.css HTTP/1.1" 302 688 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com//i/blue_bg.png HTTP/1.1" 302 687 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com//i/nav_back.gif HTTP/1.1" 302 689 "http://www.getmiro.com/" "Mozilla/4.0
[blah blah blah...]

Still four files and 33 redirects, but down to 46,506 bytes.

So far it’s all been pretty standard stuff in Apache using mod_rewrite redirects.  Apache sees a CSS sheet being called, it redirects it to S3.

RewriteRule ^/css/(.*) http://s3.getmiro.3.0.com.s3.amazonaws.com/css/$1

And then a css sheet may have a line like this:

background: url(../i/screen_dropshadow.png) -20px -36px no-repeat;

Now an observant user may note in the logs above that I switched from using Firefox to IE.  Why?  The browsers interpret the CSS differently.

Firefox interprets “../i/” relative to where the CSS style sheet is LOADED from — in our case  http://s3.getmiro.3.0.com.s3.amazonaws.com/css/.
Internet Explorer interprets “../i/” relative to where the CSS style sheet is CALLED from — in our case http://www.getmiro.com/css/.

Those familiar with unix notation know that “../i/” from “getmiro.com/css/” gets you to “getmiro.com/i/”.

4)
Now we get fancy.

In implementing S3, we have a bash script which handles the synchronization between our servers and S3.  So in that script, let’s intercept the CSS sheets, do a simple SED, and upload the modified files to a special location:

# getmiro css
# This substitutes ../i with http://s3.getmiro.3.0.com.s3.amazonaws.com/ in the getmiro css code
# and uploads them to a special directory in amazon.  This is in turn re-written by Apache to point there.
# Having the full url hard coded saves tens of thousands of redirects and gigs of bandwidth.
# It's also more efficient then "php-ifying" css to do the url substitution.

  # First, copy they css to a working directory:
    cp /data/getmiro/css/*.css /scripts/getmiro_css

  # It's safer to just modify files we know about, rather then automate finding and modifying without foreknowledge:
    sed -i 's/\.\.\/i/http:\/\/s3.getmiro.3.0.com.s3.amazonaws.com\/i/g' /scripts/getmiro_css/download-features.css
    sed -i 's/\.\.\/i/http:\/\/s3.getmiro.3.0.com.s3.amazonaws.com\/i/g' /scripts/getmiro_css/index.css
    sed -i 's/\.\.\/i/http:\/\/s3.getmiro.3.0.com.s3.amazonaws.com\/i/g' /scripts/getmiro_css/nav.css
    sed -i 's/\.\.\/i/http:\/\/s3.getmiro.3.0.com.s3.amazonaws.com\/i/g' /scripts/getmiro_css/styles.css

  # And let's upload them:
    /usr/local/s3sync/s3sync.rb -r -p -v /scripts/getmiro_css/ s3.getmiro.3.0.com:css/s3_coded/

In the background so the web developers don’t have to worry about modifying the CSS sheets to include the hard link, transforming lines like:

background: url(../i/screen_dropshadow.png) -20px -36px no-repeat;

into

background: url(s3.getmiro.3.0.com.s3.amazonaws.com/i/screen_dropshadow.png) -20px -36px no-repeat;

It’s necessary to use the pattern \.\./i/ in sed in case a developer does hard code the amazon link.  The \. means literally a period; regexes like this otherwise use a . as a wildcard to match one character, and just ../i/ would match any two characters before /i/.

In Apache, we change the redirect to this:

RewriteRule ^/css/(.*) http://s3.getmiro.3.0.com.s3.amazonaws.com/css/S3_coded/$1

With this change implemented:

"GET http://www.getmiro.com/ HTTP/1.1" 200 21781 "-" "Mozilla/4.0
 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
"GET http://www.getmiro.com//css/styles.css HTTP/1.1" 302 708 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com//css/nav.css HTTP/1.1" 302 702 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com/css/index.css HTTP/1.1" 302 706 "http://www.getmiro.com/" "Mozilla/4.0

Much better!  One file, three redirects for 23,897 bytes.  That one change represents nearly a 50% reduction in bandwidth usage on our physical servers from just the example above, and only about 1/3rd the bandwidth if we used CSS sheets being served locally even if they had the hard links to S3 on them.

5)
Finally one more tweak.

The default Apache redirect includes HTML code saying where a file has moved.  But this isn’t necessary — a web browser just needs the correct headers to tell it where to go.

So replacing the redirect to S3 we used before, we use this:

RewriteRule ^/css/(.*) /custom_messages/css_rewrite.php

This is css_rewrite.php:

<?php
/*
This rewrites css just using headers.  This saves about 300bytes per
redirect -- which saves a heck of a lot of bandwidth over time when we're doing 3
css rewrites for every page view...works out to 30+ MB / day!

Invoke by:
RewriteRule ^/css/(.*) /custom_messages/css_rewrite.php
*/

$new_server = "http://s3.getmiro.3.0.com.s3.amazonaws.com/css/s3_coded/";
$new_url = preg_replace('/^.*\//', $new_server, $_SERVER[REQUEST_URI]);

echo Header( "HTTP/1.1 301 Moved Permanently" );
echo Header( "Location: $new_url" );
?>

This produces a very minimal redirect — under 400 bytes rather then over 700 bytes.

I haven’t done a complete analysis to know if this significantly slower then a native Apache redirect; initial review shows it is not slower for any given page load.  This step would need a very, very busy site however to make a meaningful performance or cost impact.  It’s something I’m noting though, because there could be other situations this type of optimization could be useful.  Most importantly the variables Apache (or IIS) can pass to other programs like PHP.  See this link for a list of them.

"GET http://www.getmiro.com/ HTTP/1.1" 200 21781 "-" "Mozilla/4.0
 (compatible; MSIE 7.0; Windows NT 5.1; .NET CLR 1.1.4322; .NET CLR 2.0.50727)"
"GET http://www.getmiro.com//css/styles.css HTTP/1.1" 301 397 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com//css/nav.css HTTP/1.1" 301 394 "http://www.getmiro.com/" "Mozilla/4.0
"GET http://www.getmiro.com/css/index.css HTTP/1.1" 301 396 "http://www.getmiro.com/" "Mozilla/4.0

Now just one file, three redirects and 22,968.

Bottom line?

Let’s take a typical day when the getmiro.com homepage is called 20,000 times.

Scenario    Size     Total Daily      Estimated
                       Bandwidth     Daily Cost**
1           338,359        6.5GB          $4.73
2            78,705        1.5GB           1.09
3            46,506        887MB*           .65
4            23,897        455MB            .33
5            22,968        438MB            .32
* Unadjusted for Firefox's interpretation of CSS paths
** This estimate is based on purchasing enough fixed bandwidth (Mbps)
to cover our peak daily usage.  Our communication costs for our
physical servers is approximately times as much as Amazon S3 based on
actual transfers.

So without S3 or any optimization, we’d be looking at a monthly cost around $142.00.

With S3 and with all our optimization, we’re looking at a monthly cost around $54.00.

For a small non-profit, that’s a nice savings over time.

ln -s /etc/init.d/ntpd /etc/rc3.d/S99ntpd

2) /etc/ntp/step-tickers: 

time.nist.gov 
pool.ntp.org

3) In /etc/sysconfig/iptables open up the port:

# NTP is only accessible via the Gig Backbone:
-A RH-Firewall-1-INPUT -m state –state NEW -m udp -p udp –dport 123 -s 192.168.1.0/0 -j ACCEPT

4) Now (re)start iptables & ntpd

5) On internal-only servers, install ntpd. 

6) Set step-ticker to point to internal server.  If using hostname, verify /etc/hosts and/or  the /etc/resolv.conf is set properly to allow the hostname to resolve.

7) Link it to your default startup directory.

Start it

 Use a program like Ophcrack to launch a brute force attack using Rainbow Tables.  Rather then using bandwidth to download a Rainbow Table, and so you can customize the table with salts, extra characters, etc…get a RT Generator like the rtgen utility in the original RainbowCrack.

 Ah, longing for the quaint old days when you’d run L0pht all weekend to recover an administer’s password on a server you inherited…

For Data Centers, Sun offers a really good guide at: http://www.sun.com/blueprints/0501/Naming.pdf 

Every data center should at least by XY gridded — go by the floor tiles if it’s a raised floor.  When you are talking the new guy through a problem on the phone late one night, you can tell him to go to RACK B4 and look for the box labelled “SQLSRV21.”  If you have enough racks inventory becomes challenging, implement the Z axis as well.  2U = 3.5″, and if you start with “A” at the bottom, that gives you 7-1/2 feet of rack space to label A-Z for positions.

Here’s the EIA-310-D Hole Pattern, by the way:

While on the subject of labelling, do not forget to label your utilities — work with your electricians to label each outlet back to the breaker which controls it.  That way if there is an after-hours outage, it is quick to know which circuit panel needs to be checked and which breaker needs to be reset.

  Network Speed & Files…
  The general rule with ethernet is you can utilize a maximum of 80% of the bandwidth. If you have plenty of bandwidth and files are still moving slowly, look for another issue such as disk contention.
  The following calculations are done assuming the 80% efficiency factor and rounded off:
  T-1 1.54Mbps = 154KB/s, 9.25MB/m, 540MB/h
  10Mbps LAN = 1MB/s, 60MB/m, 3.5GB/h
  T-3 45Mbps = 4.5MB/s, 270MB/m, 15.8GB/h
  100Mbps = 10MB/s, 600MB/m, 35GB/h
  1Gbps = 100MB/s, 6GB/m, 351GB/h
  A handy online File Transfer Speed calculator

  To achieveve 1Gbps performance, or anything really much over 400Mbps, you need to have network cards with “TOE” (TCP Offload Engine) capability which is enabled.  Otherwise too much CPU time is taken processing network stuff at those speeds.

  USB Speed & Files…
  1.0 — 1.5Mbps
  1.1 — 12Mbps
  2.0 — 480Mbps
  So unless you’re on a 10Mbps network…
  It’s faster to use a USB 2.0 computer and transfer files via the network then suffer a USB 1.1 drive!

  Cat 3/5/6 Wiring:
  A ton of pin-outs is available at:  http://www.hardwarebook.info/
 

IP Subnetting
  Private Subnets:
  10.0.0.0 — 10.255.255.255, mask 10.0.0.0/8
  172.16.0.0 — 172.31.255.255 mask 172.16.0.0/12
  192.168.0.0 — 192.168.255.255 mask 192.168.0.0/16
  CIDR & Masks: