Archives for: April 2005

04/30/05

Permalink 12:13:53 pm, by fumanchu Email , 107 words   English (US)
Categories: IT, General

Google Bible

You know that cool preloading that Google maps does (where, when you drag the map, some of the surrounding tiles have already been loaded and instantaneously pop into view)?

We need that for an online Bible. I'm tired of getting a chapter at a time. If studylight would do that and include their Strong's numbers, I'd never go anywhere else (hint, hint).

Heck, we need that for lots of large documents.

04/27/05

Permalink 10:48:30 am, by fumanchu Email , 376 words   English (US)
Categories: IT, Dejavu

Outgrowing databases

Daniel H. Steinberg gives a summary of Adam Bosworth's recent keynote at the MySQL User's Conference 2005:

If you build an open source stack that delivers globally available information, how do you massively distribute it and cause it to scale? Bosworth said you need to limit your queries to those that can be easily implemented by everybody and those that can be handled by a single machine. This requires that your queries run at the item level. This might feel odd to those used to dealing with databases, as this means you are not likely to perform joins, aggregations, or subqueries. There is plenty of SQL that cannot be supported.

This is one of the design artifacts of Dejavu: if your domain model requires complicated joins, unions, or subqueries, it's better to refactor your model than to fight with data aggregation queries. Dejavu forces you to do so, in fact, because I didn't care to provide an object-to-SQL translation for such queries.

Refactoring may be painful, but is necessary for growth in almost any application. Best to do it up front than to be lured into fragile schemas, only to be forced to refactor or die later on.

Working backwards through the article:

Bosworth predicts that RSS 2.0 and Atom will be the lingua franca that will be used to consume all data from everywhere.

I've been thinking lately about writing a generic RSS interface for Dejavu. A simple object-property reader/writer would be a cake-walk; security would be the tough bit to design.

Imagine if you can query any data that is available anywhere in the world. Bosworth said that what this requires is a single, simple, open wire format for items. The format needs to be simple for any P programmer to deliver and any JavaScript programmer to consume.

Hm! Exactly where I've been going with Lyrica—I'm working hard to push as much as possible into Javascript on the client. The viewer will certainly be Javascript-only, with an option to traverse local files, so that the server connection isn't necessary once you've built your slideshow.

Permalink 10:09:39 am, by fumanchu Email , 286 words   English (US)
Categories: IT, Python, Dejavu, Cation

Designing from the outside in

Link: http://jtauber.com/blog/2005/04/27/designing_from_the_outside_in

...extracting a re-usable framework after the fact struck me as interesting because that's really what's happened with Leonardo. Two years ago, I wrote a little wiki-like script in Python in order to enable editing of content on jtauber.com from a browser. I then decided to expand it just over a year ago to include a blog. Now, as more features are being requested, an underlying web framework is emerging that could very well be useful outside of running a wiki or blog.

The same thing happened with Cation (a web framework) and Dejavu (an ORM). I was tasked with rewriting our core business app—two years ago, it was a procedural CGI app written in Visual Basic 4! When I rewrote the whole thing in Python, I started by isolating Cation+Dejavu into their own layer. After about six months I then separated Dejavu from Cation. In addition, I made a middle business-objects layer called "EnDue", which the final app, "Mission Control" is built on. There's also a wiki-like app called "Junct" which I built on top of Cation and Dejavu. So the tree currently looks like this:

[Cation]  [Dejavu]
    \        /
     \      /
  [EnDue][Junct]
     |
     |
[Mission Control]

I've also got "test apps" for Dejavu and EnDue (well, I'm still writing the one for EnDue...I think I'll model the business of the beard-and-stone salesman from "Life of Brian" ;) Any good names for such a business?).

Anyway, the real point I want to make (and have made before) is that I'll probably replace Cation with another web framework sometime this year...but I wouldn't have known which existing framework to pick if I hadn't written my own first.

04/24/05

Permalink 07:44:57 pm, by fumanchu Email , 157 words   English (US)
Categories: General

Fun with the multitranslator

Link: http://www.tashian.com/perl/multibabel.cgi

See if you can guess which verses (from the New Testament) begat the following:

  1. Which enters, the opening of a man does not give him the other way around for pigs, but this sae of the relative opening, that gives to shutdowns the pigs the other way around.

  2. The legend you to that the truth is left here, not a rock in another one; everything is in the game of the reduction.

  3. Loaded therefore him, where the brothers, to offer due to the tolerance of the God, their bodies have taste of alive victims, those know him and the God of the execution -- in whom it is his mental Anbetung.

  4. It would appreciate Christian and the energy of the relative Auferstehung and the friendship of the division in relative pain would know and as in the relative one it died therefore and, the method or other that takes to traverse with the Auferstehung of the inoperative men.

04/23/05

Permalink 03:45:18 pm, by fumanchu Email , 107 words   English (US)
Categories: IT

How I spent my Spring Break

I set up all this:

Network components in El Paso

  1. The old Merlin phone system, which was removed in favor of...
  2. The new Avaya digital phone system, which talks to the phones in our San Diego office over...
  3. The new T1 router, which replaces the old DSL, and serves...
  4. The LAN clients, via this hub, firewall/router, and switch, which have a couple of green cables running to...
  5. The new VPN server, which I described how to set up here and here.

04/22/05

Permalink 03:56:35 pm, by fumanchu Email , 368 words   English (US)
Categories: IT, Dejavu

Trigger unhappy

Bah. I can't design my way out of a wet paper bag when I get rushed, and I've been awfully rushed the last month or so.

The problem is update triggers. Our business (like every other) has tons of things that should happen when data changes. For example, when one of our clients decides to arrive a day earlier, that affects lots of business decisions. The people who make those decisions need to be notified, usually by email. Other, dependent business objects need to have their data updated. The very act of changing the date needs to be logged.

OK, that's not the real problem yet. The real problem is finding out when to do all of these activities. My naive first approach was to simply perform all of these side-effects whenever the FirstDate property of a MissionTrip unit changes:

def __set__(self, unit, value):
    if value != unit._properties[self.key]:
        unit._properties[self.key] = value
        self.fire_triggers(unit, value)

This is straightforward, but not very flexible. I put myself in a corner recently over this: the FirstDate is calculated based on a set of TripDate objects. When a user submits the MissionTrip web page, they might add three new TripDate objects—each one triggers a recalculation of the FirstDate property, and I ended up with three separate records in the log, along with three separate email notifications!

The local fix would be to not put a trigger on the TripDate objects, and to manually recalc the FirstDate. Wherever needed. There goes the whole point of objects.

What would be better would be a trigger that could fire at the end of the session (say, the web-page submit) instead of at the point of modification. But I can't figure out a clean way to do that. I worry that I won't remember to call session.cascadeAllTriggers() at the end of some submit handler, and I won't notice that fact for some time, since all of those activities are by nature side-effects. Not to mention the corner cases where I need to fire triggers before the end of the session.

Hmmm. I'd like something more declarative than imperative, I think.

What a messy part of domain modeling. :(

04/18/05

Permalink 11:39:36 am, by fumanchu Email , 39 words   English (US)
Categories: By By Design

Sing a song of sixpence

Alpha-numeric stamps

Can you see wich stamp is "missing" and why?

I also thought the inclusion of the "&" stamp in-between Y and Z was pretty funny—I sing the ABC song in my head every time I see it.

04/16/05

Permalink 10:55:02 pm, by fumanchu Email , 1085 words   English (US)
Categories: IT

Site-to-site ethernet bridge over OpenVPN (2 of 2)

In a previous post, I laid out step-by-step how to set up an OpenVPN server, to support roaming VPN clients. We created an ethernet bridge so that each remote client appears on the LAN just like any desktop PC in your office. We used a dedicated OpenVPN server to act like a network switch; however, each client has to install and run software in order to connect.

Now I'm going to discuss a permanent bridge, using two Linux boxes. This allows all clients on each side to see all clients on the other side. None of the clients need to run any special software. You might use this, as I did, to connect a branch office to your headquarters. Throughout this discussion, I'm going to refer to these two Linux boxes as "vpnmaster" and "vpnslave". You may call them whatever you wish when you implement this. ;)

Steps 1-9: Build vpnslave.

See the previous post, steps 1 to 9. Follow those steps without any changes. We're going to end up with the same, bridged ethernet setup on the vpnslave machine that we did with the vpnmaster.

Unlike with roaming clients, you want the same subnet in both locations. It's no good having a bridge between divergent networks. When we're done, all clients will exist on the same network. Imagine you have an unmanaged switch halfway between both locations, with very long cables in-between. ;) You need to set aside a block of IP addresses which only the computers on the slave LAN will use. For example, you might tell your DHCP server(s) at the master site to lease the addresses 192.168.0.20 to .99, and the DHCP server(s) at your slave site to lease 192.168.0.234 to .254. We'll block any DHCP broadcasts from going over the VPN, so that computers in the slave LAN will only talk to their local DHCP servers.

Step 10: Copy and edit the firewall script.

cp /usr/src/openvpn-2.0_rc21/sample-config-files/firewall.sh /etc/openvpn/firewall.sh
cd /etc/openvpn
vi firewall.sh
  • Change the value of PRIVATE to your local subnet. Yours might be 192.168.0.0/24. Again, use the same network for the slave site that you use at the master site.
  • After iptables -A INPUT -p udp --dport 1194 -j ACCEPT
    add the line iptables -A INPUT -p udp --dport 4444:4460 -j ACCEPT
    or whatever port range you're going to use for your clients.
  • If the example firewall script has NAT MASQUERADING turned on (last line), comment it out in your new copy. (If you're wondering, we can't simply use NAT to bridge two different networks, because the broadcasts would not then be routed across the bridge).
  • Add the following lines to the script. These turn off DHCP broadcasts across the slave's bridge. You probably do not want to do this wholesale DROP on vpnmaster, because it's probably serving roaming clients who want DHCP.

iptables -A FORWARD -p tcp --dport 67:68 -j DROP
iptables -A FORWARD -p udp --dport 67:68 -j DROP
iptables -A INPUT -p tcp --dport 67:68 -j DROP
iptables -A INPUT -p udp --dport 67:68 -j DROP

Step 11: Create /etc/openvpn/opentuns.sh.

At this point, you only need one item in the TAPS list. I used "0" to make "tap0". You may use vpnslave to serve its own remote clients, as well, but we're not going to describe that here.

Step 12: Create /etc/openvpn/bridge.conf.

This file provides config information for the permanent bridge on vpnslave. It's a little bit different from common.conf on vpnmaster.

remote openvpn.aminus.org
dev tap0
port 1194
secret /etc/openvpn/static.key
ping 10
verb 5
comp-lzo

Step 13: Create an init script.

Create the file /etc/init.d/openvpn, to run everything at startup:

#!/bin/bash

/etc/openvpn/firewall.sh
/etc/openvpn/opentuns.sh

/usr/local/sbin/openvpn --config /etc/openvpn/bridge.conf --daemon tap0

exit 0

Remember to make this file executable, using chmod.

Notice we're only running one instance of openvpn. Again, if you want to serve additional, roaming clients from vpnslave, feel free. But I'd recommend doing that after the permanent pipe is up and running.

In /etc/rc2.d through rc5.d, add symlinks to the file you just created. The "S number" you supply in the filename will determine the order in which your startup script is run (lowest first). I picked 19 a bit arbitrarily, based on other scripts in the rc folders.

ln -s /etc/init.d/openvpn S19openvpn

Step 14: Copy the static.key from vpnmaster to vpnslave.

Use scp, sftp, a floppy, a thumbdrive...whatever it takes. But keep it secret. Keep it safe.

Step 15: There is no Step 15.

Step 16: Test OpenVPN.

Make sure that OpenVPN is running on vpnmaster. On vpnslave, type the following. Note that, for the most part, we are doing by hand what your init.d script does. We do it by hand so we can see the output at each step, in case something goes wrong:

cd /etc/openvpn
./firewall.sh
./opentuns.sh

/usr/local/sbin/openvpn --config /etc/openvpn/bridge.conf --dev tap0 --port 1194

Notice we do not supply the daemon argument to openvpn on the command line. This means your terminal will now be occupied running openvpn. You should start seeing traffic display in the openvpn process: it'll start spitting out w's and r's as it reads and writes data from vpnmaster.

Alt-F2 to another tty and login again so you can test the connection. Try accessing resources on the LAN (ping remote boxes by IP and by name). If something goes wrong, fix it before proceeding to the next step.

If the vpnslave can ping machines on the master LAN (over the VPN), then move to another computer on the slave LAN and see if you get the same results. In some network setups, your default gateway may need a new static route, directing 192.168.0.0 packets to eth1 on vpnslave. If you've got a single gateway and a single switch or hub, you shouldn't have to do this.

Step 17: Test your startup script.

Kill the openvpn process from step 16 using Ctrl-C. Reboot vpnslave. Test the connection again (from vpnslave). If that works, test again from another computer on the slave LAN.


If all that works, you're done! Unplug the monitor, keyboard, and mouse from vpnslave and tuck it away somewhere. Come back in 5 years when the hard drive starts to make loud banging noises.

Comments and corrections are certainly welcome.

Permalink 06:52:41 pm, by fumanchu Email , 1511 words   English (US)
Categories: IT

Site-to-site bridged ethernet using OpenVPN (1 of 2)

Our headquarters is in San Diego, CA, where I live and work. The last few days, however, I've been working in El Paso, TX, where we have a branch office. I had three tasks this week:

  1. Help "turn up" (enable) the new T1 line to the El Paso office. I was there when it was installed 3 weeks ago, but I had to leave before it could be turned on.
  2. Bridge the LAN in the El Paso office with the LAN in San Diego, placing them on the same subnet.
  3. Install a new Avaya phone router for the El Paso phones (made by Lucent), which then are linked through the VPN to San Diego, again appearing as if both networks are contiguous.

I'm going to provide a step-by-step for #2, bridged ethernet over a VPN. But I'll only lay out one-half of it now: the server side in San Diego, with roaming clients. The next post will discuss the El Paso side, since that has its own complexities.

To accomplish this, I used OpenVPN, an open-source, userspace VPN solution made by James Yonan. Give him some money if you can—this is the greatest networking tool since tcpdump.

Step 1: Get a server.

You don't have to dedicate one solely to OpenVPN, but you probably should. I used a $400 Wal-Mart PC (with no OS). It needs two NIC's.

Step 2: Install Debian Linux.

I used Sarge, which is currently the "testing" release, but will Very Soon Now become the "stable" release. I used the sweet new debian-installer with a 2.6 kernel, as sysadmin guru Greg Folkert advised.

Step 3: Connect the hardware.

Give eth0 a static IP. The configs will be easier if eth0 is the external NIC. You should have one cable hooked into your LAN switch/hub, and the other cable should be on the other side of your broadband router or firewall. It should look like this:

Diagram of VPN network

You don't have to have a T1, but you need multiple, static IP's (from your ISP). The firewall should already have one, and eth0 needs another. Also, be aware that your T1 box may have its own already (and don't use your broadcast address, either ;) ).

Step 4: Install LZO compression and the lzo headers.

apt-get install liblzo1
apt-get install liblzo-dev

Step 5: Install OpenSSL and its headers.

apt-get install openssl
apt-get install libssl-dev

Step 6: Install OpenVPN.

cd /usr/src
wget ??

Replace ?? with the URL of the OpenVPN sources. Go to the OpenVPN website to find out what the URL of the most recent release is. Modify the following commands to match whatever release you end up downloading. Also, do your best to use the same version on both the server and the clients.

gunzip openvpn-2.0_rc21.tar.gz
tar xvf openvpn-2.0_rc21.tar
cd openvpn-2.0_rc21
./configure --with-ssl-headers=/usr/local/ssl/include/ --with-ssl-lib=/usr/local/ssl/lib/
make
make install

Step 7: Get more tools.

apt-get install bridge-utils
apt-get install tcpdump

Step 8: Turn on IP forwarding.

Edit /etc/network/options, set ip_forward=yes

Step 9: Configure the tun device.

mkdir /dev/net
mknod /dev/net/tun c 10 200

Step 10: Copy and edit the firewall script.

cp /usr/src/openvpn-2.0_rc21/sample-config-files/firewall.sh /etc/openvpn/firewall.sh
cd /etc/openvpn
vi firewall.sh
  • Change the value of PRIVATE to your local subnet. Yours might be 192.168.0.0/24
  • After iptables -A INPUT -p udp --dport 1194 -j ACCEPT
    add the line iptables -A INPUT -p udp --dport 4444:4449 -j ACCEPT
    or whatever port range you're going to use for your clients.
  • If the example firewall script has NAT MASQUERADING turned on (last line), comment it out in your new copy.

Step 11: Create /etc/openvpn/opentuns.sh.

#!/bin/bash

/usr/sbin/brctl addbr br0
/usr/sbin/brctl addif br0 eth1

TAPS="public rbre elpaso"

for name in $TAPS
do
    /usr/local/sbin/openvpn --mktun --dev tap$name
    /usr/sbin/brctl addif br0 tap$name
done

ifconfig eth1 0.0.0.0 promisc up

for name in $TAPS
do
    ifconfig tap$name 0.0.0.0 promisc up
done

ifconfig br0 192.168.0.235 netmask 255.255.255.0 broadcast 192.168.0.255

exit 0

Add a new name to the TAPS list for each tap (port) you wish to create.

The brctl commands make a new ethernet bridge, and then bind eth1 to that bridge. This means br0 is now acting like an unmanaged switch on your LAN, and eth1 is "plugged into" that switch. When each tap is bound to the same bridge, they become participants on that switch, and will receive any traffic which passes over eth1 (they're all "promiscuous", which means they will listen for traffic even if it's not directed specifically at them). The last ifconfig line in the script makes our bridge behave like a managed switch by giving it an IP address.

Remember to make opentuns.sh executable, using chmod.

Step 12: Create /etc/openvpn/common.conf.

This file provides a common config for all taps.

secret /etc/openvpn/static.key
float
ping 10
verb 5
comp-lzo
persist-tun
persist-remote-ip
persist-key

Step 13: Create an init script.

Create the file /etc/init.d/openvpn, to run everything at startup:

#!/bin/bash

/etc/openvpn/firewall.sh
/etc/openvpn/opentuns.sh

VPN="/usr/local/sbin/openvpn --config /etc/openvpn/common.conf"

$VPN --dev tappublic --port 1194 --daemon tappublic
$VPN --dev taprbre   --port 4444 --daemon taprbre
$VPN --dev tapelpaso --port 4445 --daemon tapelpaso

exit 0

Sorry, but you have to add a new line for each tap you create. This maps the port to the tap.

Remember to make this file executable, using chmod.

In /etc/rc2.d through rc5.d, add symlinks to the file you just created. The "S number" you supply in the filename will determine the order in which your startup script is run (lowest first). I picked 19 a bit arbitrarily, based on other scripts in the rc folders.

ln -s ../init.d/openvpn S19openvpn

Step 14: Make a static key on the server.

cd /etc/openvpn (mkdir the folder if necessary)
openvpn --genkey --secret static.key
chmod go-rwx static.key

Step 15: Install OpenVPN on the client(s).

All of my roaming clients are Windows 2k or XP; download the installer on each client. Each client needs a .conf file (the name doesn't matter) which matches the one on the server (for most items):

remote openvpn.aminus.org
port 4444

dev tap
dev-node my-tap

secret static.key
ping 10
verb 5
comp-lzo

mute 10

The remote parameter needs to point to eth0 on your server. My domain registrar has a nice DNS tool, so I added an A (host) record for openvpn.aminus.org and pointed it to the IP which I gave eth0. If you don't care to use DNS, you can just type the IP of eth0.

On Windows clients, remember to name the new network connection "my-tap", or whatever name you use in the client config.

Copy the static.key from the server into the /config folder on each client.

Step 16: Test OpenVPN.

On the server, type the following. Note that, for the most part, we are doing by hand what your init.d script does. We do it by hand so we can see the output at each step, in case something goes wrong:

cd /etc/openvpn
./firewall.sh
./opentuns.sh

/usr/local/sbin/openvpn --config /etc/openvpn/common.conf --dev tappublic --port 1194

Notice we do not supply the daemon argument to openvpn on the command line. This means your terminal will now be occupied running openvpn. If you need to check anything on the server while testing, Alt-F2 to another tty and login again.

Now start OpenVPN on a client; make sure the port is the same (1194, in our example). On the server side, you should start seeing traffic display in the openvpn process: it'll start spitting out w's and r's as it reads and writes data from the client. On the client side, if you start openvpn manually (in Windows, right-click on your .conf file and select "Start OpenVPN on this file") you should also see the connection as it happens. From the client, try accessing resources on the LAN (ping IPs, browse network shares, or access an internal website). If something goes wrong, fix it before proceeding to the next step.

The first thing that will most likely go wrong: you've got your server LAN running on the 192.168.0.x subnet, and you're running the client at home, also on a 192.168.0.x subnet. This doesn't work well, because your client doesn't know whether to send local packets through the tunnel or not. If your home was running on, say, 192.168.15.x, then there would be no conflict. Change your work or home subnet to something else.

Step 17: Test your startup script.

Kill the openvpn process from step 16 using Ctrl-C. Reboot the server. Test a client again, manually. If that works, test the OpenVPNService (on Windows clients).


This should work for as many roaming clients as you like. OpenVPN 2.0 is supposed to have a way to set this up without having to specify a separate port per client; I simply haven't looked at that yet. I will when we get too many roaming clients.

Next time, we'll talk about a permanent bridge between two networks.

Permalink 11:31:22 am, by fumanchu Email , 750 words   English (US)
Categories: IT, Python

The hymn microformat (HMML?)

Dave Warnock has been talking about a new microformat for hymns, and has chosen an initial set of primitives that are surprisingly close to what I've had in Lyrica for some time now. I wrote the first version of Lyrica a couple of years ago, and the whole application seems rather baroque to me now, for many different reasons:

  1. The tools to put together a slideshow are IE-only, because
  2. The app runs serverless, which means it needs an ActiveX object to write local files.
  3. The make-a-slideshow UI sucks in many ways. I don't like the load/save flow in particular.
  4. Although the song files themselves are quite minimal, there's too much layout information contained in them—the order of verse-chorus-etc shouldn't be hard-coded by the order it appears in the song file. I mistakenly told David that I never repeat choruses—turns out I have on a few occasions to get around the limitations of my design.

There are, however, several design choices I still like:

  1. The song files are pretty minimal, and approach Dave's concept of an open format for hymns. If you're looking for a name for it, Dave, try HMML: Hymn Microformat Markup Language. "Himmel" is German for "Heaven" ;) (among other meanings).
  2. The javascript for running the slideshow is nice and simple, and extensible with custom plugins per style.
  3. The CSS is simple, partly because of the microformat.

Dave talked about using Eric Meyer's S5 for the transform from bare content to slideshows. I still don't like S5's one-style-per-show design—Lyrica allows my church to apply different styles to each song pretty quickly, without hand-editing any files. So I think I'll revamp it again here Real Soon Now. I'd like to see HMML become transparent enough that both S5 and Lyrica operate on it with little fuss, but I simply don't see S5 having a model which fits Sunday morning song lyrics. Sermons, maybe, but not songs.

What I'll focus on in a redesign:

  1. IE-only has to go, which means a server will have to handle the song-editing and slideshow-composition persistence. I resisted requiring a server for a while, since churches tend to lag behind the tech curve. However, if a stable microformat can be reached, I can probably get the server component down to a very small standalone Python webserver, which can run on localhost:8080 for example. With an appropriate, minimal REST spec, the server side could be written in a variety of languages.
  2. Most of the slideshow-composition tools will get some nice AJAX flows. No more modal dialogs. Hooray!
  3. I'll look at moving the lyric flow into Javascript instead of the current 'slide' divs.

No, better yet, I'll move the lyric flow out to the operator. Hymns may be sung the same way every time in every church, but modern praise music is certainly not. One church might sing Verse-Chorus-Verse-Chorus while another might sing VVCV-Bridge-CC-Tag. Many praise bands replay lyric blocks "as the Spirit leads", both interminably and randomly, it would seem. The slideshow operator, therefore, needs both a "standard" flow template and tools to modify that flow on the fly.

I strongly believe, therefore, that the microformat should neither repeat lyric blocks, nor should it contain flow markers inline like Dave's "repeatchorus" paragraphs. The flow of blocks should be pushed out to metadata at worst, and out to the operator at best. A metdata solution might be a meta name='default_flow' content='verse1 chorus verse2 chorus' tag or something similar. An operator tool might be a set of JS key bindings available while a given song is displayed. Perhaps a nav overlay could be displayed while the layout decision is being made? Arrow-keys could follow the default flow specified in the meta tag.

Finally, a microformat that moved flow out of the main content would cut out an enormous amount of wasted duplication. I already have some songs with multiple files, because the words are slightly different between versions. A central repository of songs would probably see an explosion of versions, resulting primarily from differing lyric arrangement.

A flow-free microformat would also make writing a HMML editor much easier.

Hmmm... much to think about.

Permalink 10:46:07 am, by fumanchu Email , 110 words   English (US)
Categories: IT

Various definitions of "correct functioning"

From Coding Horror

Unit tests don't guarantee correct functioning of a program. I think it's unreasonable to expect them to. But writing unit tests does guarantee that the developer has considered, however briefly, these truly difficult testing questions. And that's clearly a step in the right direction.

It may be a step in the right direction. Here's the next step: turn those "developer considerations" into functional requirements, and then use those to define "correct functioning". Then you'll have your guarantee. It isn't the unreasonable guarantee you started with, but it's a heck of a lot better than the second gurantee you offered, Jeff. ;)

Permalink 10:17:00 am, by fumanchu Email , 52 words   English (US)
Categories: By By Design

So that's why Americans abandoned Brit spellings

When you spell words "the American way", you don't have to work so hard parsing (and pronouncing) compounds like "Centrelink" (the name of an Australian government agency). That is, you won't ever pronounce "Centerlink" like /cen-treh-link/.

So the question is, is this by by design?

04/11/05

Permalink 12:14:06 am, by fumanchu Email , 37 words   English (US)
Categories: General, Photography

A black day for John Dempsey

His 40th birthday, that is. Here's the obligatory pictures of:

The surprise.

John being surprised

The cake.

40th Birthday Cake

The confetti. Lots of confetti.

Confetti (4 pics) Boy spinning with confetti in his hair

The crowd.

Panorama of messy interior

The Toast.

A toast to the old man

Here's to the next 40, Stubaby. I want to be you when I grow up.

04/10/05

Permalink 10:28:08 pm, by fumanchu Email , 63 words   English (US)
Categories: Photography

The moon over Balboa Park

The matte appearance of the dark side of the moon, despite the fairly bright crescent, struck me while driving home tonight.

NW Moon over Balboa Park

These two were taken with different exposure times:

Two moons, different exposures

The moon sets a bit north of west, and was almost perfectly in line with the airport. Here's a plane ascending after takeoff:

Plane ascending in front of moon, long exposure

And another, after having approached and then turned toward the north:

Plane moving away from moon, long exposure

04/09/05

Permalink 01:21:13 pm, by fumanchu Email , 38 words   English (US)
Categories: General

Help me to remember the saints of old

Link: http://onlyaninkling.blogspot.com/2005/03/yahoo-reflection-on-way-to-mexico.html

Found via IceRocket. Here's a feed for "amor ministries", if anyone is interested.

Permalink 11:20:19 am, by fumanchu Email , 1036 words   English (US)
Categories: IT

Orchestrating the stack with small IT budgets

I just had a Blog-Aha! moment; I realized that I should be blogging about the decisions we've made as a small, tech-heavy non-profit. On Thursday, Wendy (my team leader) suggested I prepare a talk for CMA on using Python to glue disparate systems together. Then today, I was pointed to Geoffrey Moore's keynote at Software 2004 (watch the video with the PDF slides in the background) via IT Conversations.

Moore's talk wasn't directed to me—he was talking to developers of "Enterprise IT" offerings. But as the head of IT at a small nonprofit, I'm certainly as interested in "Orchestrating the Stack" for my company as any of those developers could be. One slide, in particular, caught my eye this morning; he presented a nice overview of "the stack", and made fairly obvious choices about which vendor was the 800-pound gorilla in each category, and which vendors are the chimps, or challengers. Note that not every layer in the stack has a gorilla:

CategoryGorillaChimps
Consulting ServicesIBM, Accenture, EDS, Infosys
Transaction ApplicationsSAPOracle, Siebel, Peoplesoft
Analytical ApplicationsCognos, SAS, Bus Obj
Communications & CollaborationMicrosoftIBM
Application InfrastructureBEA, IBM, Microsoft
Utility InfrastructureCA, IBM, Veritas, Symantec
DatabaseOracleIBM, Microsoft
Server Operating SystemMicrosoft, Linux, Sun, HP
MainframesIBM
ServersDell, HP, Sun, IBM
StorageEMCHitachi, HP, Dell, Fujitsu
NetworkCiscoJuniper, Foundry, Extreme
MicroprocessorIntelAMD

Point Numero Uno (more to come in future posts)

As a small business, we find we can't buy a gorilla in every category. The lower layers, perhaps, as they become commoditized. Our processors are mostly Intel, our firewall is Cisco, and most of our servers are Dell (but notice there's no gorilla in that server layer).

But we can't afford Oracle in the database layer, for example. Not just because the product itself is expensive, but because that gorilla relies on support in the other layers—we could buy an install of Oracle, perhaps, but not the Consulting (layer) it would need to be used effectively, nor the Transaction or Analytical Applications built for Oracle, nor a lot of the lower layers. We're not big enough to buy Sun servers or IBM mainframes or any of the Storage layer stuff; our storage layer is still Western Digital and Maxtor.

We certainly can't buy SAP. Our IT budget (excluding salaries and capital exp) is ½ of 1 percent of our total $5M expense budget. And I still call us "tech-heavy"? Well, yes, because the value that our IT staff of 2 provides to the company far outweighs the costs. So if we can't buy Oracle and SAP, how do we "orchestrate the stack"?

Geoffrey Moore told his audience to "marginalize the chimps" in their own layer and to "marginalize layers not entered". Because of our small budget, I am the provider of some of those layers, and I have the same strategy. I provide these layers for my company:

  • Consulting Services. I am the consultant, not because I know best, but because we cannot afford the person who knows best. No, scratch that. I am the person that knows our company best.
  • Transaction Applications. We have a unique business: building homes for the poor. We can't just buy off-the-shelf software built for construction companies, for nonprofit or volunteer organizations, or for churches. We need a custom solution, so I built one.
  • Analytical Applications. We're almost that big. But not quite. But I'm working on it.

In these layers, I am the gorilla, and the chimps simply don't care about little ol' us and aren't trying. They're marginalized by cost, scale, and specialization.

As the gorilla in those three layers, I'm working aggressively to marginalize the other layers—not so I can enter them, but to make them into commodities. Open-source development has allowed me to do this with the middle layers:

  • Application Infrastructure. This is currently owned by Python. Python makes my application layers wonderfully cross-platform at almost every layer below it. This layer is also owned by HTML/CSS, in my world.
  • Utility Infrastructure. Needs some shoring up at our company, but I'd like this to become Ryan Gwillim's space. It's nicely orthogonal to most of the concerns I have in the app layer. To name a few products/vendors, we use TrendMicro, spamassassin, and OpenVPN in this layer.
  • Database. Currently MS Access, MS SQL Server, and a little MySQL. However, Python will allow me to move from Access to Postgres later this year, again at the flip of a switch (the SQL Server is locked in by a third-party app).
  • Operating System. Microsoft and Debian share this space. The presence of MS is due more to lock-in with other layers (Exchange, Blackberry, Blackbaud) than any other reason.

Servers and below are such commodities that I don't deal with those decisions much anymore. It's all good enough, fast enough, and cheap enough.

Notice that I didn't mention "Communications & Collaboration" in either of the above lists. That's the one area where we're stuck with a gorilla: Microsoft Exchange (although it's not enough for us, so there's a Linux box running exim in front of it). This blog is eroding some of that space, but it's the calendaring and the integration with Blackberry's and other devices that's keeping Exchange entrenched. We're also investigating MS Project, but I secretly hope Basecamp will win that spot. This is the one layer I worry about, because Microsoft is certainly doing everything it can to leverage its dominance in this layer, in an attempt to dominate the other layers. We cannot afford to let this happen. I don't mean that in the anything-but-Microsoft sense. We cannot afford it, period—our pocketbook can't afford it.

So, that's where we are. In the future, I'll discuss some more detailed aspects of those choices, and where we expect to change as we grow.

Permalink 10:25:23 am, by admin Email , 92 words   English (US)
Categories: General

And I attend a Presbyterian church?

My results from the denominational quiz via 42

  1. Pentecostal/Charismatic/Assemblies of God (100%)
  2. Anabaptist (Mennonite/Quaker etc.) (98%)
  3. Church of Christ/Campbellite (83%)
  4. Baptist (non-Calvinistic)/Plymouth Brethren/Fundamentalist (81%)
  5. Methodist/Wesleyan/Nazarene (75%)
  6. Seventh-Day Adventist (61%)
  7. Baptist (Reformed/Particular/Calvinistic) (53%)
  8. Congregational/United Church of Christ (44%)
  9. Anglican/Episcopal/Church of England (36%)
  10. Lutheran (23%)
  11. Eastern Orthodox (21%)
  12. Presbyterian/Reformed (20%)
  13. Roman Catholic (12%)

Those Presbyterians must be draggin' me down--I'm pretty sure I went to a "Campbellite" Christian College... ;)

04/08/05

Permalink 03:57:53 pm, by fumanchu Email , 0 words   English (US)
Categories: IT

Proof: computers only know what you tell them

Who has 192.168.0.130? Tell 192.168.0.130

Permalink 02:18:41 pm, by fumanchu Email , 670 words   English (US)
Categories: IT

Flying the Gauges

Yesterday, I posted the following on IWETHEY:

So I decided to install MRTG yesterday to monitor our Cisco PIX. Very nice tool so far, and oddly enough, very handy today. The PIX was bouncing along at < 1% usage until 1:45 PM today, when it jumped to about 36 percent, where it has stayed ever since (long after everyone has gone home):

MRTG traffic graph with a flat line

Notice you read the graph from right-to-left (left is most recent data).

I was of course curious about which client on my LAN could be passing so much gas. Or *accepting*, rather, since the green is inbound data. That's when I realized I had no way to tell, or if I do, I don't know about it. I've got an SNMP trapper, which shows me each request in both directions across the PIX, but doesn't show me total bytes. I've run Ethereal inside, but it doesn't show me _all_ the traffic on the LAN [what it *did* show me was that I had a chatty UPnP webcam, which I've since silenced, but that wasn't the pipe hog].

My two big 48-port switches aren't managed. Stupid me--I'll know better next time.

So where do I go from here to find the offending NIC?

As usual, I got some very informative and helpful replies. The IWETHEY Group Mind™ at its best again. :)

Unfortunately, nothing helped. I started to "do the unplug dance", shutting off computers. I ran network monitors on the others. Ethereal helped me find a user running Gnutella, another with a worm that's emailing the world, and a webcam that didn't need to be running UPnP. I found users with both a Cat5 cable and WiFi+VPN on at the same time. I learned, to my surprise, that our VOIP switches were on our LAN switch, when I had thought otherwise. But I couldn't find the machine that was "chewing up our T1".

So I wasn't wasting my day, but I certainly wasn't reaching the goal I had set: to change that flat green line. Finally, I started thinking, "OK, something's wrong with the graph itself". I decided it might be a Windows thing. Hm.

Beep. Beep. Change the KVM switch to one of our Linux boxes.

apt-get install mrtg No problem there.

Hm. Better tell the PIX to feed SNMP to the Linux box. Beep. Beep. Change the KVM to the machine with the PIX console.

show snmp

Oops. No snmp entry for the MRTG instance I had been running for two days. Dangit. I had put it in there, but hadn't committed it to the flash memory, and at some point the PIX got rebooted and lost the entry. Added it back, wrote it to memory, flushed the MRTG log and restarted it. Fifteen minutes later I have a reasonable traffic pattern again, not clearing 5% of the PIX's max speed. Like it should be.

More normal traffic graph

Are you ready for the worst part? The MRTG FAQ says:

My graphs show a flat line during an outage. Why ?

Well, the short answer is that when an SNMP query goes out and a response doesn't come back, MRTG has to assume something to put in the graph, and by default it assumes that the last answer we got back is probably closer to the truth than zero. This assumption is not perfect (as you have noticed). It's a trade-off that happens to fail during a total outage.

But that's not the worst part. The worst part is that I had read that two days ago, and made a conscious effort to remember it. And I did remember it, after I had fixed the problem. But somehow, that piece of information never made it out of my brain's memory into my brain's decision-making area.

Unless you can apply it at the right time, all info is just trivia. Its potential uses are irrelevant.

04/06/05

Permalink 10:21:38 pm, by fumanchu Email , 218 words   English (US)
Categories: General

Another day, another bus...

In our last episode, dear readers, we left our intrepid band stranded in Mexico. To make a long story short, they got to their worksites just fine every day, using the Mexican bus line they had originally hired for that purpose.

However, they still had to get back across the border, and the tour bus they came in wasn't about to re-enter Mexico (said bus having been reclaimed on Wednesday, and yes, a fine was paid). So we arranged for a fourth bus company, Mexicoach, to take the group from camp back across the border. They met at Carl's Jr and (happy chance) I was there at the same time, taking a different group in. But that's another story. So, here's a pic of the Mexicoach handing passengers off to the tour bus. Ron, the driver of the U.S. bus, is in the blue shirt near the front of the bus—a bit frustrated by the events, of course, but a patient and understanding fellow, a class act.

Mexicoach and U.S. Tour Bus

Astute readers may question why there's a police car with its lights on in the foreground...I have no comment.

If they do it again, they'll probably use Mexicoach to transfer people at the border in both directions.

Permalink 04:53:04 pm, by fumanchu Email , 49 words   English (US)
Categories: General

Plumb crazy

A house in Juarez with a preexisting toilet

We don't install plumbing or electricity in the houses we build. The only problem with that approach is that, once in a while, the house recipients will decide to beat us to the punch. This homeowner decided to install a toilet and shower drain before we laid the foundation.

Permalink 04:46:31 pm, by fumanchu Email , 20 words   English (US)
Categories: General

Cutest. Hammer. Ever.

Roberta the hammer and her owner, Tiffany

This is a hammer. Its name is Roberta. Its owner's name is Tiffany. Hammer, Tiffany, hammer! Don't scratch that paint!

Permalink 12:47:03 am, by fumanchu Email , 261 words   English (US)
Categories: General, Photography

Rare pics of Plaustrum Exsputuminum

This rare species of van is usually found in large groups, exhibiting mutually-beneficial social behaviors such as the "queue" when resources become scarce. Leadership is well-defined by skin color, with the larger, darker males taking precedence in directing the periodic migrations of the herd.

Line of parked vans

Note the coordinated display of bright red markings, signifying the vans' acquiescence to the dominant leader, as he signals the end of the day's journey.

These vans are quite docile under normal circumstances. Once separated from the safety of the other vans, however, the Plaustrum Exsputuminum (or "spitting van") quickly becomes agitated. The front, side, and rear doors are puffed out to make the beast appear larger and more threatening, although they are already impressive at nearly 5 tons. In some cases, the hood is also extended. Often, the animal will explosively eject its contents at an attacker, as you can see in this second photograph:

15-passenger van with doors open and luggage strewn about

If you see a van with its doors or hood open, DO NOT APPROACH IT. Adult vans can accurately spray their luggage onto an attacker over 15 feet away. In addition, the frightened van may close its doors onto an unsuspecting victim's exposed hand, leg, or head. If you encounter a van in the wild, you should:

  • Make yourself look bigger.
  • Yell loudly but in a low voice. RrrrrrRrrrrrRrrrr.
  • Slowly walk backwards away from the animal. Vans do not navigate well in reverse.
  • Pick up small children right away.

March/April and June through August are the best times to observe these gentle giants, as they head south, seeking campgrounds throughout northern Mexico.

Permalink 12:01:03 am, by fumanchu Email , 6 words   English (US)
Categories: Photography

Covenant Presbyterian Church: The Basement

Covenant Presbyterian Church basement

Landsakes, but this church has assets.

04/05/05

Permalink 11:40:20 pm, by fumanchu Email , 87 words   English (US)
Categories: Photography

Coffee table photos

Just playin' around. Love the Canon A95.

Love how the reflected checkerboard seems to bend with the Diet Coke swoosh. It really doesn't except by coincidence; the arc of the swoosh and the arc of the can's cylinder happen to coincide for a time.

Something about the bold bar of light, reflecting off the can, really drew my eye to this one. Coupled with the "drain" concept (and in-line with it), it stands in huge contrast to the regular checkerboard pattern of the rest of the piece.

Permalink 11:06:55 pm, by fumanchu Email , 21 words   English (US)
Categories: General

Hooray for regulation

Link: http://seattletimes.nwsource.com/html/travel/2002232032_webpassportsinfo06.html

Knowing our government, it'll actually be the end of 2009, but still... this is going to cut down our business something fierce.

Permalink 01:03:08 pm, by fumanchu Email , 32 words   English (US)
Categories: Photography

Caption Contest

I'm too tired to give a good caption, so I'll be boring and say: Spring Break is really tiring around here. If you've got a better caption, post it in a comment!

04/04/05

Permalink 08:49:45 pm, by fumanchu Email , 171 words   English (US)
Categories: Dejavu

Bug in dejavu.storage.db

There was a bug in dejavu.storage.db until today (revision 69). You should probably upgrade from SVN if you're using dejavu.

The bug manifests whenever you have a logic.Expression that has a function call which cannot be represented in SQL. For example, I hit it with the Expression: lambda x, **kw: x.FirstDate < datetime.date(kw['Year'], 12, 1).

There are two parts to explaining why this went unnoticed for so long. First, all of my uses of datetime.date had been constants so far. When they are made into Expressions, the early binder turns the whole .date constructor call into a LOAD_CONST. Consequently, I've never written a dispatcher for the datetime.date constructor, which is a function. So db.SQLDecompiler.visit_CALL_FUNCTION had no concrete function object to call, and nowhere to dispatch to. When both of those most-common cases fell through, the decompiler stack should have gotten a cannot_represent entry appended to it; instead, that entry was overwriting the top-of-stack. Ugly and oh-so-fun to track down.

04/03/05

Permalink 12:13:09 am, by fumanchu Email , 74 words   English (US)
Categories: General

Sushi night at Howie's

Howard and Victoria, both excellent chefs, hosted a sushi night at their home in Imperial Beach in honor of Tracy, a field staff member at Amor who is moving on. Here's a big, ugly stitched view from their balcony:

View of Pacific from Majors home

We made a ton of sushi, more than we could eat. Here are some of the creations. I'm sure Ryan is green with envy since he's not in town. ;)

3 kinds of Sushi

Bye Tracy. :( Come back real soon.

Tracy and Howie

04/02/05

Permalink 10:51:04 am, by fumanchu Email , 500 words   English (US)
Categories: General

Passion is nice. Obedience is better.

From Rick Warren:

Another word for heart is passion. There are certain subjects that you feel deeply passionate about and others that you couldn’t care less about. Some experiences turn you on and capture your attention, while others turn you off or bore you to tears. These reveal the nature of your heart.

When you were growing up you may have discovered that you were intensely interested in some subjects that no one else in your family cared about. Where did those interests come from? They came from God!

God had a purpose in giving you these inborn interests. Your emotional heartbeat is the second key to understanding your shape for service. Don’t ignore your interests. Consider how they might be used for God’s glory. There is a reason that you love to do these things.

Repeatedly, the Bible says to "serve the Lord with all your heart." God wants you to serve him passionately, not dutifully. People rarely excel at tasks they don't enjoy doing or feel passionate about. God wants you to use your natural interests to serve him and others.

Riiiight. Tell that to Moses, or Jonah, or Jesus. Those don't sound like "natural interests" to me.

The worst thing about such a "passion-dependent" theology of ministry is that, when you do encounter difficulty, you're going to excuse yourself by saying, "oh, well, that's not my passion," as if we lived in a consumer-driven ministry-market.

When the revolution comes and you're in front of the firing squad, will you comfort those around you? Minister to your enemies? Preach to them? Seek to overcome injustice? Or is that not your passion?

When you are sent by the Lord to pass by the homeless woman being attacked on the street, will you intercede on her behalf? Physically? Spend time, money, and energy getting her medical attention? Find her a job? Or is that not your passion?

Even if God knows "your purpose" before you were born, you don't. Don't pretend you do.

"...remember all the commandments of the LORD, so as to do them and not follow after your own heart and your own eyes, after which you played the harlot, so that you may remember to do all My commandments and be holy to your God." Num 15:39-40

04/01/05

Permalink 10:23:55 am, by fumanchu Email , 69 words   English (US)
Categories: Python

Got my 2nd edition Python Cookbook, today

Had to leaf through 757 pages to find my name ;) but it's there. Now if I can just figure out what I contributed to that recipe...

Ah. Here's my comment on Stephen's excellent recipe. Hm. I still don't see how any of my comment got into the printed recipe. Oh, well. I'm just happy to have a complimentary copy of the Cookbook. ;)

April 2005
Sun Mon Tue Wed Thu Fri Sat
 << < Current> >>
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30

Search

The requested Blog doesn't exist any more!

XML Feeds

powered by b2evolution