Steve Glendinning

After hours fighting with WCCP, i’ve given up and implemented the simpler solution: policy-based routing.

WCCP is a cisco protocol for managing web caches. It’s really quite slick, as it only forwards requests to the cache(s) when they are alive (and sending “i am here” messages to the router). If the cache service fails, the router passes web requests directly through. WCCP also automatically handles some of the fiddlier configuration, such as not mangling requests from the cache itself. Unfortunately I couldn’t get it to work.

My Cisco 877 is currently running the latest 12.4T IOS (12.4(15)T1). Some of the web guides I found suggested “known working” versions of 12.3 or 12.4 mainline IOS, but only 12.4T versions of IOS are available for the 877. This leaves a lot of variables, I might open a TAC case and get Cisco on the job.

Policy-based routing works, but it doesn’t gracefully handle cache failure like WCCP. On the Cisco 877:

no ip cef

access-list 101 deny tcp host 10.0.20.1 any eq www
access-list 101 permit tcp any any eq www

route-map proxy-redir permit 10
match ip address 101
set ip next-hop 10.0.20.1

interface Vlan1
ip policy route-map proxy-redir
ip route-cache policy

Where 10.0.20.1 is the IP address of the squid webcache.

This only works when I turn off CEF (no ip cef). When CEF is enabled, the first packet of the TCP connection (SYN) is forwarded from router to webcache, the webcache replies directly to the client (SYN|ACK), but the third packet from client (ACK) does not get forwarded by the router to the webcache. All connections time out.

When the policy-based routing is process switched the forwarding works as expected. All packets arrive at the webcache and the caching is transparent as expected. Fast-switched policy-based routing (ip route-cache policy) also works, which is an improvement on process-based, but the optimal solution would be CEF-based. I have a Cisco TAC case open to investigate this.

On the Linux 2.6 (debian Etch) squid server:

# Disable rp_filters
echo 0 > /proc/sys/net/ipv4/conf/default/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/all/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/lo/rp_filter
echo 0 > /proc/sys/net/ipv4/conf/eth0/rp_filter

# transparent webcaching
iptables -t nat -A PREROUTING -s 10.0.20.0/24 -d ! 10.0.20.0/24 -p tcp –dport 80 -j DNAT –to-destination 10.0.20.1:3128

10.0.20.0/24 is the subnet to cache, and 10.0.20.1 is the IP address of the webcache.

That’s it. HTTP requests are transparently forwarded to the squid server and cached.

I found these resources helpful when trying to get WCCP working:

• Squid and WCCP (November 2005)
• Squid FAQ entry on WCCP
• Squid Cache Transparency and WCCP (September 2001)
• WCCPv2 GRE with 2.6 on Linux

And these were useful for policy based routing:

• Squid FAQ entry on Interception Caching packet redirection with Cisco routers using policy routing (NON WCCP)
• Squid FAQ entry on Linux 2.0.33 and Cisco policy-routing
• Cisco guide to configuring policy-based routing

I just finished upgrading my Debian Etch fileserver from 2×200GB IDE disks to 2×500GB SATA disks. I managed to keep the server running for nearly the entire time, by failing and hot-adding disks to the RAID-1 arrays. If I had room in the case for more than two disks it would have been even easier.

Here is the configuration BEFORE:

• /dev/hda partitioned into hda1 (10GB), hda2 (1GB), hda3 (175GB)
• /dev/hdc partitioned into hdc1 (10GB), hdc2 (1GB), hdc3 (175GB)
• RAID-1 array md0 composed of hda1 and hdc1, mounted as /
• RAID-1 array md1 composed of hda2 and hdc2, mounted as swap
• RAID-1 array md2 composed of hda3 and hdc3, mounted as /home

I started the ball rolling by failing one partition from each RAID array:

mdadm –fail /dev/md0 /dev/hdc1
mdadm –fail /dev/md1 /dev/hdc2
mdadm –fail /dev/md2 /dev/hdc3

Then I powered down the server, disconnected and removed hdc and added a new 500GB SATA disk to the SATA PCI card. It booted up fine with all three RAID arrays degraded. I used fdisk to partition the new SATA disk (/dev/sda) with identical sized partitions 1 and 2, and with the third partition taking up the remainder of the disk. I set all partition types to fd (linux raid auto-detect):

• sda1 (10GB), sda2 (1GB), sda3 (454GB)

Then one at a time I hot-added these partitions to the running RAID arrays. This causes a background reconstruction, so it’s worth waiting for each to finish before starting the next:

mdadm –add /dev/md0 /dev/sda1
mdadm –add /dev/md1 /dev/sda2
mdadm –add /dev/md2 /dev/sda3

When all three were completely synced (cat /proc/mdstat to see the progress), I edited /etc/mdadm/mdadm.conf to change all references from /dev/hdcx to /dev/sdax. I then re-built the initramfs so it knew how to start the arrays at boot time:

update-initramfs -k all -c -t

I then powered down the server again, removed the last IDE disk (hda) and added the second SATA disk (sdb). At this point the system is unbootable, so I started from a rescue CD (actually the Debian Etch netinst cd, starting with the “rescue” boot option). Once I got a command prompt (Alt-F2 and Alt-F3 virtual consoles), I installed grub:

mount /dev/sda1 /mnt
chroot /mnt /bin/bash
nano /boot/grub/device.map

I edited the device.map so it looked like this:

(hd0) /dev/sda
(hd1) /dev/sdb

Then installed grub on the first SATA disk:

grub-install /dev/sda

I rebooted and grub succesfully booted the server. As expected, all RAID arrays were in degraded mode. I used fdisk to re-partition the second SATA disk to match the first, then hot-added the mirrors to the RAID arrays (waiting for each re-sync to complete before starting the next):

mdadm –add /dev/md0 /dev/sdb1
mdadm –add /dev/md1 /dev/sdb2
mdadm –add /dev/md2 /dev/sdb3

Then I edited /etc/mdadm/mdadm.conf to update the partitions to (for example) sda1,sdb1. I re-build the initramfs again as above, and rebooted to test everything booted up cleanly. Also I checked /proc/mdstat after the reboot to check all arrays were fully functional.

So now the new disks are installed, but there’s no extra storage available because the ext3 partition is still set to the old size! I rebooted into single user mode, unmounted /home, then used resize2fs to expand the filesystem to use the whole partition:

e2fsck -f /dev/md2
resize2fs /dev/md2

One reboot later and voila, 455GB usable in /home.

Then I followed this guide to install grub on the second RAID disk in a bootable way.

Here are some fantastic blog posts and resources I’ve come across in the last week:

• “Madduck” posted two extremely detailed posts on using the git revision control tool. Here is the first, and here is the second. They’re specific to debian packaging, but they’re the kind of worked examples I find extremely useful.  I think git has a lot of potential, especially with its excellent svn integration.
• Chris Lamb posted his top 10 shell shortcuts (bash, zsh etc) . I only knew a few of these, they could save me some keyboard time.
• ScottGu posted yet another useful collection of .net resources. Particularly handy is the C# coding standard guidelines from IDesign.

Nokia have released an updated version of their Mail For Exchange application which officially supports the N73. The 1.5 version explicitly blocked installation on N73, while the older 1.3 release worked but was unsupported.

If you run a Microsoft Exchange server, this is a must-have application as it enables seamless sync of contacts and calendar as well as full push email. And it’s completely free.

The e-series blog and Darla Mack have more information on the release and its new features, and you can download it here: http://businesssoftware.nokia.com/mail_for_exchange_downloads.php

So I’ve decided to experiment with Google Adsense, and so far I’m very impressed! After a week for Google to approve my application (why?), the welcome email arrived. I installed the Adsense Manager plugin for Wordpress, made a slight tweak to my theme, and as if by magic… adverts!

One good thing this forced me to do is clone the default theme. Now I can start tweaking it a bit more severely, expect breakage!

The Debian package for Wordpress in Etch is version 2.0. Lenny (the current testing distribution) has version 2.2, which has quite a few improvements. Here’s how I installed this testing package without upsetting the rest of my stable system.

Wordpress is a php package, so it’s architecture independant and has hardly any dependencies. In fact the only dependency that can’t be satisfied from stable is libphp-phpmailer: wordpress 2.2 needs a newer version of this library than shipped with etch. It’s possible to manually download those two packages from the testing distribution and install them manually, but there’s a better way.

Add the testing distribution to your /etc/apt/sources.list so it looks like this (the first two should already be in there):

deb http://ftp.uk.debian.org/debian/ etch main contrib non-free
deb http://security.debian.org/ etch/updates main contrib non-free
deb http://ftp.uk.debian.org/debian/ testing main

Then use apt-pinning to disable the testing distribution for all packages except the ones you want. Edit the /etc/apt/preferences file (you may have to create it if it doesn’t already exist) so it looks like this:

Package: *
Pin: release a=stable
Pin-Priority: 700

Package: *
Pin: release a=testing
Pin-Priority: -1

Package: wordpress
Pin: release a=testing
Pin-Priority: 800

Package: libphp-phpmailer
Pin: release a=testing
Pin-Priority: 800

The magic part of this is the preference of “-1″ for the testing distribution, which removes its packages from the available list. If you put a positive preference such as 500 (which is lower than the stable preference of 700), stable will still “win” for most packages but you’ll see some that aren’t in stable. The preference of 800 is assigned to the two packages we *do* want, and this makes them preferred over stable.

Now install wordpress 2.2:

apt-get update
apt-get install wordpress

If you already have wordpress 2.0 installed, “apt-get update; apt-get upgrade” will upgrade your 2.0 copy instead.

This technique can be used to cherry pick other packages from testing into a stable base system, but only if they don’t have significant dependencies (such as a newer c library version). In this case, backports.org is your friend.