The Hand of FuManChu

I set up all this:

1. The old Merlin phone system, which was removed in favor of...
2. The new Avaya digital phone system, which talks to the phones in our San Diego office over...
3. The new T1 router, which replaces the old DSL, and serves...
4. The LAN clients, via this hub, firewall/router, and switch, which have a couple of green cables running to...
5. The new VPN server, which I described how to set up here and here.

Bah. I can't design my way out of a wet paper bag when I get rushed, and I've been awfully rushed the last month or so.

The problem is update triggers. Our business (like every other) has tons of things that should happen when data changes. For example, when one of our clients decides to arrive a day earlier, that affects lots of business decisions. The people who make those decisions need to be notified, usually by email. Other, dependent business objects need to have their data updated. The very act of changing the date needs to be logged.

OK, that's not the real problem yet. The real problem is finding out when to do all of these activities. My naive first approach was to simply perform all of these side-effects whenever the FirstDate property of a MissionTrip unit changes:

def __set__(self, unit, value):
    if value != unit._properties[self.key]:
        unit._properties[self.key] = value
        self.fire_triggers(unit, value)

This is straightforward, but not very flexible. I put myself in a corner recently over this: the FirstDate is calculated based on a set of TripDate objects. When a user submits the MissionTrip web page, they might add three new TripDate objects—each one triggers a recalculation of the FirstDate property, and I ended up with three separate records in the log, along with three separate email notifications!

The local fix would be to not put a trigger on the TripDate objects, and to manually recalc the FirstDate. Wherever needed. There goes the whole point of objects.

What would be better would be a trigger that could fire at the end of the session (say, the web-page submit) instead of at the point of modification. But I can't figure out a clean way to do that. I worry that I won't remember to call session.cascadeAllTriggers() at the end of some submit handler, and I won't notice that fact for some time, since all of those activities are by nature side-effects. Not to mention the corner cases where I need to fire triggers before the end of the session.

Hmmm. I'd like something more declarative than imperative, I think.

What a messy part of domain modeling.

Can you see wich stamp is "missing" and why?

I also thought the inclusion of the "&" stamp in-between Y and Z was pretty funny—I sing the ABC song in my head every time I see it.

In a previous post, I laid out step-by-step how to set up an OpenVPN server, to support roaming VPN clients. We created an ethernet bridge so that each remote client appears on the LAN just like any desktop PC in your office. We used a dedicated OpenVPN server to act like a network switch; however, each client has to install and run software in order to connect.

Now I'm going to discuss a permanent bridge, using two Linux boxes. This allows all clients on each side to see all clients on the other side. None of the clients need to run any special software. You might use this, as I did, to connect a branch office to your headquarters. Throughout this discussion, I'm going to refer to these two Linux boxes as "vpnmaster" and "vpnslave". You may call them whatever you wish when you implement this.

Steps 1-9: Build vpnslave.

See the previous post, steps 1 to 9. Follow those steps without any changes. We're going to end up with the same, bridged ethernet setup on the vpnslave machine that we did with the vpnmaster.

Unlike with roaming clients, you want the same subnet in both locations. It's no good having a bridge between divergent networks. When we're done, all clients will exist on the same network. Imagine you have an unmanaged switch halfway between both locations, with very long cables in-between. You need to set aside a block of IP addresses which only the computers on the slave LAN will use. For example, you might tell your DHCP server(s) at the master site to lease the addresses 192.168.0.20 to .99, and the DHCP server(s) at your slave site to lease 192.168.0.234 to .254. We'll block any DHCP broadcasts from going over the VPN, so that computers in the slave LAN will only talk to their local DHCP servers.

Step 10: Copy and edit the firewall script.

cp /usr/src/openvpn-2.0_rc21/sample-config-files/firewall.sh /etc/openvpn/firewall.sh
cd /etc/openvpn
vi firewall.sh

• Change the value of PRIVATE to your local subnet. Yours might be 192.168.0.0/24. Again, use the same network for the slave site that you use at the master site.
• After iptables -A INPUT -p udp --dport 1194 -j ACCEPT
  add the line iptables -A INPUT -p udp --dport 4444:4460 -j ACCEPT
  or whatever port range you're going to use for your clients.
• If the example firewall script has NAT MASQUERADING turned on (last line), comment it out in your new copy. (If you're wondering, we can't simply use NAT to bridge two different networks, because the broadcasts would not then be routed across the bridge).
• Add the following lines to the script. These turn off DHCP broadcasts across the slave's bridge. You probably do not want to do this wholesale DROP on vpnmaster, because it's probably serving roaming clients who want DHCP.

iptables -A FORWARD -p tcp --dport 67:68 -j DROP
iptables -A FORWARD -p udp --dport 67:68 -j DROP
iptables -A INPUT -p tcp --dport 67:68 -j DROP
iptables -A INPUT -p udp --dport 67:68 -j DROP

Step 11: Create /etc/openvpn/opentuns.sh.

At this point, you only need one item in the TAPS list. I used "0" to make "tap0". You may use vpnslave to serve its own remote clients, as well, but we're not going to describe that here.

Step 12: Create /etc/openvpn/bridge.conf.

This file provides config information for the permanent bridge on vpnslave. It's a little bit different from common.conf on vpnmaster.

remote openvpn.aminus.org
dev tap0
port 1194
secret /etc/openvpn/static.key
ping 10
verb 5
comp-lzo

Step 13: Create an init script.

Create the file /etc/init.d/openvpn, to run everything at startup:

#!/bin/bash

/etc/openvpn/firewall.sh
/etc/openvpn/opentuns.sh

/usr/local/sbin/openvpn --config /etc/openvpn/bridge.conf --daemon tap0

exit 0

Remember to make this file executable, using chmod.

Notice we're only running one instance of openvpn. Again, if you want to serve additional, roaming clients from vpnslave, feel free. But I'd recommend doing that after the permanent pipe is up and running.

In /etc/rc2.d through rc5.d, add symlinks to the file you just created. The "S number" you supply in the filename will determine the order in which your startup script is run (lowest first). I picked 19 a bit arbitrarily, based on other scripts in the rc folders.

ln -s /etc/init.d/openvpn S19openvpn

Step 14: Copy the static.key from vpnmaster to vpnslave.

Use scp, sftp, a floppy, a thumbdrive...whatever it takes. But keep it secret. Keep it safe.

Step 15: There is no Step 15.

Step 16: Test OpenVPN.

Make sure that OpenVPN is running on vpnmaster. On vpnslave, type the following. Note that, for the most part, we are doing by hand what your init.d script does. We do it by hand so we can see the output at each step, in case something goes wrong:

cd /etc/openvpn
./firewall.sh
./opentuns.sh

/usr/local/sbin/openvpn --config /etc/openvpn/bridge.conf --dev tap0 --port 1194

Notice we do not supply the daemon argument to openvpn on the command line. This means your terminal will now be occupied running openvpn. You should start seeing traffic display in the openvpn process: it'll start spitting out w's and r's as it reads and writes data from vpnmaster.

Alt-F2 to another tty and login again so you can test the connection. Try accessing resources on the LAN (ping remote boxes by IP and by name). If something goes wrong, fix it before proceeding to the next step.

If the vpnslave can ping machines on the master LAN (over the VPN), then move to another computer on the slave LAN and see if you get the same results. In some network setups, your default gateway may need a new static route, directing 192.168.0.0 packets to eth1 on vpnslave. If you've got a single gateway and a single switch or hub, you shouldn't have to do this.

Step 17: Test your startup script.

Kill the openvpn process from step 16 using Ctrl-C. Reboot vpnslave. Test the connection again (from vpnslave). If that works, test again from another computer on the slave LAN.

---

If all that works, you're done! Unplug the monitor, keyboard, and mouse from vpnslave and tuck it away somewhere. Come back in 5 years when the hard drive starts to make loud banging noises.

Comments and corrections are certainly welcome.

Our headquarters is in San Diego, CA, where I live and work. The last few days, however, I've been working in El Paso, TX, where we have a branch office. I had three tasks this week:

1. Help "turn up" (enable) the new T1 line to the El Paso office. I was there when it was installed 3 weeks ago, but I had to leave before it could be turned on.
2. Bridge the LAN in the El Paso office with the LAN in San Diego, placing them on the same subnet.
3. Install a new Avaya phone router for the El Paso phones (made by Lucent), which then are linked through the VPN to San Diego, again appearing as if both networks are contiguous.

I'm going to provide a step-by-step for #2, bridged ethernet over a VPN. But I'll only lay out one-half of it now: the server side in San Diego, with roaming clients. The next post will discuss the El Paso side, since that has its own complexities.

To accomplish this, I used OpenVPN, an open-source, userspace VPN solution made by James Yonan. Give him some money if you can—this is the greatest networking tool since tcpdump.

Step 1: Get a server.

You don't have to dedicate one solely to OpenVPN, but you probably should. I used a $400 Wal-Mart PC (with no OS). It needs two NIC's.

Step 2: Install Debian Linux.

I used Sarge, which is currently the "testing" release, but will Very Soon Now become the "stable" release. I used the sweet new debian-installer with a 2.6 kernel, as sysadmin guru Greg Folkert advised.

Step 3: Connect the hardware.

Give eth0 a static IP. The configs will be easier if eth0 is the external NIC. You should have one cable hooked into your LAN switch/hub, and the other cable should be on the other side of your broadband router or firewall. It should look like this:

You don't have to have a T1, but you need multiple, static IP's (from your ISP). The firewall should already have one, and eth0 needs another. Also, be aware that your T1 box may have its own already (and don't use your broadcast address, either ).

Step 4: Install LZO compression and the lzo headers.

apt-get install liblzo1
apt-get install liblzo-dev

Step 5: Install OpenSSL and its headers.

apt-get install openssl
apt-get install libssl-dev

Step 6: Install OpenVPN.

cd /usr/src
wget ??

Replace ?? with the URL of the OpenVPN sources. Go to the OpenVPN website to find out what the URL of the most recent release is. Modify the following commands to match whatever release you end up downloading. Also, do your best to use the same version on both the server and the clients.

gunzip openvpn-2.0_rc21.tar.gz
tar xvf openvpn-2.0_rc21.tar
cd openvpn-2.0_rc21
./configure --with-ssl-headers=/usr/local/ssl/include/ --with-ssl-lib=/usr/local/ssl/lib/
make
make install

Step 7: Get more tools.

apt-get install bridge-utils
apt-get install tcpdump

Step 8: Turn on IP forwarding.

Edit /etc/network/options, set ip_forward=yes

Step 9: Configure the tun device.

mkdir /dev/net
mknod /dev/net/tun c 10 200

Step 10: Copy and edit the firewall script.

cp /usr/src/openvpn-2.0_rc21/sample-config-files/firewall.sh /etc/openvpn/firewall.sh
cd /etc/openvpn
vi firewall.sh

• Change the value of PRIVATE to your local subnet. Yours might be 192.168.0.0/24
• After iptables -A INPUT -p udp --dport 1194 -j ACCEPT
  add the line iptables -A INPUT -p udp --dport 4444:4449 -j ACCEPT
  or whatever port range you're going to use for your clients.
• If the example firewall script has NAT MASQUERADING turned on (last line), comment it out in your new copy.

Step 11: Create /etc/openvpn/opentuns.sh.

#!/bin/bash

/usr/sbin/brctl addbr br0
/usr/sbin/brctl addif br0 eth1

TAPS="public rbre elpaso"

for name in $TAPS
do
    /usr/local/sbin/openvpn --mktun --dev tap$name
    /usr/sbin/brctl addif br0 tap$name
done

ifconfig eth1 0.0.0.0 promisc up

for name in $TAPS
do
    ifconfig tap$name 0.0.0.0 promisc up
done

ifconfig br0 192.168.0.235 netmask 255.255.255.0 broadcast 192.168.0.255

exit 0

Add a new name to the TAPS list for each tap (port) you wish to create.

The brctl commands make a new ethernet bridge, and then bind eth1 to that bridge. This means br0 is now acting like an unmanaged switch on your LAN, and eth1 is "plugged into" that switch. When each tap is bound to the same bridge, they become participants on that switch, and will receive any traffic which passes over eth1 (they're all "promiscuous", which means they will listen for traffic even if it's not directed specifically at them). The last ifconfig line in the script makes our bridge behave like a managed switch by giving it an IP address.

Remember to make opentuns.sh executable, using chmod.

Step 12: Create /etc/openvpn/common.conf.

This file provides a common config for all taps.

secret /etc/openvpn/static.key
float
ping 10
verb 5
comp-lzo
persist-tun
persist-remote-ip
persist-key

Step 13: Create an init script.

Create the file /etc/init.d/openvpn, to run everything at startup:

#!/bin/bash

/etc/openvpn/firewall.sh
/etc/openvpn/opentuns.sh

VPN="/usr/local/sbin/openvpn --config /etc/openvpn/common.conf"

$VPN --dev tappublic --port 1194 --daemon tappublic
$VPN --dev taprbre   --port 4444 --daemon taprbre
$VPN --dev tapelpaso --port 4445 --daemon tapelpaso

exit 0

Sorry, but you have to add a new line for each tap you create. This maps the port to the tap.

Remember to make this file executable, using chmod.

In /etc/rc2.d through rc5.d, add symlinks to the file you just created. The "S number" you supply in the filename will determine the order in which your startup script is run (lowest first). I picked 19 a bit arbitrarily, based on other scripts in the rc folders.

ln -s ../init.d/openvpn S19openvpn

Step 14: Make a static key on the server.

cd /etc/openvpn (mkdir the folder if necessary)
openvpn --genkey --secret static.key
chmod go-rwx static.key

Step 15: Install OpenVPN on the client(s).

All of my roaming clients are Windows 2k or XP; download the installer on each client. Each client needs a .conf file (the name doesn't matter) which matches the one on the server (for most items):

remote openvpn.aminus.org
port 4444

dev tap
dev-node my-tap

secret static.key
ping 10
verb 5
comp-lzo

mute 10

The remote parameter needs to point to eth0 on your server. My domain registrar has a nice DNS tool, so I added an A (host) record for openvpn.aminus.org and pointed it to the IP which I gave eth0. If you don't care to use DNS, you can just type the IP of eth0.

On Windows clients, remember to name the new network connection "my-tap", or whatever name you use in the client config.

Copy the static.key from the server into the /config folder on each client.

Step 16: Test OpenVPN.

On the server, type the following. Note that, for the most part, we are doing by hand what your init.d script does. We do it by hand so we can see the output at each step, in case something goes wrong:

cd /etc/openvpn
./firewall.sh
./opentuns.sh

/usr/local/sbin/openvpn --config /etc/openvpn/common.conf --dev tappublic --port 1194

Notice we do not supply the daemon argument to openvpn on the command line. This means your terminal will now be occupied running openvpn. If you need to check anything on the server while testing, Alt-F2 to another tty and login again.

Now start OpenVPN on a client; make sure the port is the same (1194, in our example). On the server side, you should start seeing traffic display in the openvpn process: it'll start spitting out w's and r's as it reads and writes data from the client. On the client side, if you start openvpn manually (in Windows, right-click on your .conf file and select "Start OpenVPN on this file") you should also see the connection as it happens. From the client, try accessing resources on the LAN (ping IPs, browse network shares, or access an internal website). If something goes wrong, fix it before proceeding to the next step.

The first thing that will most likely go wrong: you've got your server LAN running on the 192.168.0.x subnet, and you're running the client at home, also on a 192.168.0.x subnet. This doesn't work well, because your client doesn't know whether to send local packets through the tunnel or not. If your home was running on, say, 192.168.15.x, then there would be no conflict. Change your work or home subnet to something else.

Step 17: Test your startup script.

Kill the openvpn process from step 16 using Ctrl-C. Reboot the server. Test a client again, manually. If that works, test the OpenVPNService (on Windows clients).

---

This should work for as many roaming clients as you like. OpenVPN 2.0 is supposed to have a way to set this up without having to specify a separate port per client; I simply haven't looked at that yet. I will when we get too many roaming clients.

Next time, we'll talk about a permanent bridge between two networks.

Dave Warnock has been talking about a new microformat for hymns, and has chosen an initial set of primitives that are surprisingly close to what I've had in Lyrica for some time now. I wrote the first version of Lyrica a couple of years ago, and the whole application seems rather baroque to me now, for many different reasons:

1. The tools to put together a slideshow are IE-only, because
2. The app runs serverless, which means it needs an ActiveX object to write local files.
3. The make-a-slideshow UI sucks in many ways. I don't like the load/save flow in particular.
4. Although the song files themselves are quite minimal, there's too much layout information contained in them—the order of verse-chorus-etc shouldn't be hard-coded by the order it appears in the song file. I mistakenly told David that I never repeat choruses—turns out I have on a few occasions to get around the limitations of my design.

There are, however, several design choices I still like:

1. The song files are pretty minimal, and approach Dave's concept of an open format for hymns. If you're looking for a name for it, Dave, try HMML: Hymn Microformat Markup Language. "Himmel" is German for "Heaven" (among other meanings).
2. The javascript for running the slideshow is nice and simple, and extensible with custom plugins per style.
3. The CSS is simple, partly because of the microformat.

Dave talked about using Eric Meyer's S5 for the transform from bare content to slideshows. I still don't like S5's one-style-per-show design—Lyrica allows my church to apply different styles to each song pretty quickly, without hand-editing any files. So I think I'll revamp it again here Real Soon Now. I'd like to see HMML become transparent enough that both S5 and Lyrica operate on it with little fuss, but I simply don't see S5 having a model which fits Sunday morning song lyrics. Sermons, maybe, but not songs.

What I'll focus on in a redesign:

1. IE-only has to go, which means a server will have to handle the song-editing and slideshow-composition persistence. I resisted requiring a server for a while, since churches tend to lag behind the tech curve. However, if a stable microformat can be reached, I can probably get the server component down to a very small standalone Python webserver, which can run on localhost:8080 for example. With an appropriate, minimal REST spec, the server side could be written in a variety of languages.
2. Most of the slideshow-composition tools will get some nice AJAX flows. No more modal dialogs. Hooray!
3. I'll look at moving the lyric flow into Javascript instead of the current 'slide' divs.

No, better yet, I'll move the lyric flow out to the operator. Hymns may be sung the same way every time in every church, but modern praise music is certainly not. One church might sing Verse-Chorus-Verse-Chorus while another might sing VVCV-Bridge-CC-Tag. Many praise bands replay lyric blocks "as the Spirit leads", both interminably and randomly, it would seem. The slideshow operator, therefore, needs both a "standard" flow template and tools to modify that flow on the fly.

I strongly believe, therefore, that the microformat should neither repeat lyric blocks, nor should it contain flow markers inline like Dave's "repeatchorus" paragraphs. The flow of blocks should be pushed out to metadata at worst, and out to the operator at best. A metdata solution might be a meta name='default_flow' content='verse1 chorus verse2 chorus' tag or something similar. An operator tool might be a set of JS key bindings available while a given song is displayed. Perhaps a nav overlay could be displayed while the layout decision is being made? Arrow-keys could follow the default flow specified in the meta tag.

Finally, a microformat that moved flow out of the main content would cut out an enormous amount of wasted duplication. I already have some songs with multiple files, because the words are slightly different between versions. A central repository of songs would probably see an explosion of versions, resulting primarily from differing lyric arrangement.

A flow-free microformat would also make writing a HMML editor much easier.

Hmmm... much to think about.

From Coding Horror

Unit tests don't guarantee correct functioning of a program. I think it's unreasonable to expect them to. But writing unit tests does guarantee that the developer has considered, however briefly, these truly difficult testing questions. And that's clearly a step in the right direction.

It may be a step in the right direction. Here's the next step: turn those "developer considerations" into functional requirements, and then use those to define "correct functioning". Then you'll have your guarantee. It isn't the unreasonable guarantee you started with, but it's a heck of a lot better than the second gurantee you offered, Jeff.

When you spell words "the American way", you don't have to work so hard parsing (and pronouncing) compounds like "Centrelink" (the name of an Australian government agency). That is, you won't ever pronounce "Centerlink" like /cen-treh-link/.

So the question is, is this by by design?

His 40th birthday, that is. Here's the obligatory pictures of:

The surprise.

The cake.

The confetti. Lots of confetti.

The crowd.

The Toast.

Here's to the next 40, Stubaby. I want to be you when I grow up.

The matte appearance of the dark side of the moon, despite the fairly bright crescent, struck me while driving home tonight.

These two were taken with different exposure times:

The moon sets a bit north of west, and was almost perfectly in line with the airport. Here's a plane ascending after takeoff:

And another, after having approached and then turned toward the north: