Archives for: July 2005

07/30/05

Permalink 01:58:46 pm, by fumanchu Email , 1199 words   English (US)
Categories: CherryPy, WSGI

Plugin madness

Glyph is talking about a complete refactoring of divmod, and mentions:

At every point in implementing this system we have known whether to fuse a component together because we'd built unnecessary additional complexity into previous systems, and where to use a plug-in architecture because we'd needed to inject ugly code into the middle of a monolithic routine.

As a result, where our architecture was heavily monolithic before, now it is almost entirely composed of plugins. It is so plugin-happy, in fact, that there is a database with Service plugins in it, which activate when the database is started from twistd; it contains its own configuration, including port-numbers, so nothing need live in a text configuration file.

Plugins are great because they facilitate customization: you can make small changes in system behavior with small changes in your code. An architecture that is "monolithic", to use Glyph's term, is one where small changes in system behavior require large changes in your code.

CherryPy 2.1 has a system of Filters (both built-in and user-provided), which act as plugins. As each HTTP request is processed, there are a few fixed points where the Request processor searches for registered Filter methods and gives up control to them. The Filter then must either return control to the Request processor, or raise a control-flow exception, like NotFound, RequestHandled, or HTTPRedirect. Here's the guts of the Request processor (the run method of the Request class) itself:

def run(self):
    """Process the Request."""
    try:
        try:
            applyFilters('onStartResource')

            try:
                self.processRequestHeaders()

                applyFilters('beforeRequestBody')
                if cherrypy.request.processRequestBody:
                    self.processRequestBody()

                applyFilters('beforeMain')
                if cherrypy.response.body is None:
                    main()

                applyFilters('beforeFinalize')
                finalize()
            except cherrypy.RequestHandled:
                pass
            except cherrypy.HTTPRedirect, inst:
                # For an HTTPRedirect, we don't go through the regular
                # mechanism: we return the redirect immediately
                inst.set_response()
                finalize()
        finally:
            applyFilters('onEndResource')
    except cherrypy.NotFound:
        cherrypy.response.status = 404
        handleError(sys.exc_info())
    except:
        handleError(sys.exc_info())

It's decent; that is, it's fairly clean and understandable IMO. But it's quite limited in two very important ways:

  1. There are only 7 points at which customization is possible [beforeErrorResponse and afterErrorResponse are found inside handleError]. Any filters which require the core to release control at multiple points have to do some fancy dancing to coordinate state between those applyFilters calls. Any filters which require additional control points are out of luck—their only recourse is to handle the remainder of the process themselves (probably with generous cut-and-paste from the CP core) and raise RequestHandled.

  2. Certain core processes are locked in time. For example, processRequestHeaders gets the "path" from the first line of the HTTP request; however, the Filters themselves are supposed to be dependent upon the current path! Therefore, for example, onStartResource filters must always be global ("/").

  3. (Slightly unrelated: the builtin "request" filters get run before any user-defined filters, and vice-versa for "response" filters. This needs a fix).

I tried to ameliorate some of these issues in the short term a) by making a Request class (which 2.0 didn't have); that might become subclassable someday, and b) by keeping a lot of logic out of the Request class, placing it instead in module-level global functions (which people can then call as needed).

A couple of weeks ago, I joked that maybe main (which calls the user's page handler) and finalize (which cleans up the response status, headers, and body) should themselves become filters, and it wasn't entirely a joke. When I look at CherryPy as-is, I see not one, but three separate API's.

The first, simplest one is for application developers, and includes:

  1. The cherrypy.root hierarchy.
  2. Page handler methods, including the spec for index and default methods, and the "exposed" attribute.
  3. Passing CGI params (etc) to page handler methods as keyword args.
  4. Expecting response content to be passed back to the core via "return" or "yield" statements.

The second API is quite different. It assumes a much higher level of competency, and is both more powerful and more complicated. I see it as mostly useful for those writing frameworks or libraries on top of CherryPy, although many "normal" app developers will end up using some of this interface. It includes:

  1. Filters. Creating, organizing, maintaining.
  2. Changing status, headers, or body via cherrypy.response.
  3. For that matter, almost anything involving cherrypy.request or .response: cookies and sessions, path inspection, HTTP-method dispatching, etc.

Finally, there's a third "API", which CherryPy supports quite well, but in a different fashion. There are a number of people who will run into, say, the filter limitations I outlined above, and will customize their copy of CherryPy to do what they want. One of my design goals has always been to make this easier by making the core insanely simple. There has been a lot of work done to keep the various components isolated, preferring a data-driven approach, centered around the cherrypy.request and .response objects, mostly; that is, your filter or site-specific customization can do whatever it likes, as long as it sets valid response.status, .headers, and .body before returning control to the HTTP server.

I don't see anything fundamentally wrong with having "different API's". It would be nice for some items from the "middle layer" to become more easily-accessible to the "shallow layer"; a lot of that can be done with clever wrapper classes, customized for specific situations. But I think it's quite all right to have a separation between a clean, simple, limited interface and a more powerful, but more complicated, interface behind that, to be used when needed.

However, I'd like to see the "lowest layer" unstick itself a bit more yet. That Request.run method, in particular, is far too frozen at the moment—I'd like to see it become the default processor, with an easy way to override some or all of it. I think that would free up CherryPy developers to better manage the current web-application space, which continues to change quickly as new ideas and technologies roll in. [Turning on my World Domination Mode for a moment, it might also allow a small, focused CherryPy core to become the backplane for several of the existing Python web frameworks, especially as more of them begin to support WSGI.]

Part of the reason there is a Filter specification at all is to shield CherryPy application deployers; they now have an interface for plugging in various components, that is simpler than, say, subclassing Request. For example, a deployer can decide to gzip their HTTP responses with a single line in their config file. In addition, the developer of that app need not be aware that this is being done. It's a "freebie" from his or her point of view.

But what if one could write one's own Request.run in such a way that that became easier than using config files? If that were possible, almost all of the overhead of the Filter architecture could be removed completely. In addition, developers and deployers could share total control over the request process, rather than the limited, dare-I-say "clunky" process we have now with filters. I think that with CherryPy 2.1, we're halfway there, and it won't take much work to make it happen in an elegant and powerful fashion.

07/29/05

Permalink 01:33:21 am, by fumanchu Email , 400 words   English (US)
Categories: General

Ichi, ni, sashimi

FUN college group meeting tonight—I taught the guys how to make sushi! There was some initial skepticism:

A hesitant Corey and Fred

...but after I showed them how it was done:

Me stuffing a big maki chunk into my mouth

...Fred tried some ginger:

Fred eating a piece of gari (ginger)

...and Corey tried some salmon roe by itself X(

Corey gagging on roe

...and they dove in:

Me, Corey, and Fred making sushi

Here's a rainbow roll Fred made:

Rainbow sushi roll

(Thanks for the pictures, Linda!)

After an hour or more of making all kinds of sushi, we cleaned up a bit, then talked about "friends and enemies". Reading Matthew 5, how serious is Jesus when he tells you to "turn the other cheek" to your enemy? Who is your enemy? Do you have any? Or (like many American Christians that I know), have you designed your life so well that you only spend time with your friends? I think I do, quite often. We find being hit on the left cheek so distasteful that we never expose our right side in the first place, and therefore never have to make the hard decision to "go the extra mile".

We also talked about those who are neither friends (our ingroup) or our enemies (our outgroup), but "nobodies". Jesus called them the poor, the lame, and the sick, and in Luke 14, says "when you give a banquet, don't invite your friends". But he doesn't say "invite your enemies"; instead, you should invite the nobodies. In Luke 16, he mentions the "shrewd manager" who loses money to gain friends (I wonder how many people have done the exact opposite). So we talked a bit about what it means to sacrifice, for whom, and why. And who will be mad when you do? Probably your friends ("a man's enemies will be members of his own household").

It seems Jesus is calling people to step out of their social boundaries, in effect joining a new "ingroup": the church. And just like MySpace's Tom, you have to be friends with the creator to be a member. ;) However, there's an additional responsibility to continually bring people—from your old ingroup (friends, family, social class), from your enemies, and from the nobodies—into the new group. I hate to say it, but I see a lot of churches fall down at this point; they neglect and isolate themselves from one or more of those groups. I'm glad I have the chance on a daily basis to serve the poor in Mexico...

07/21/05

Permalink 12:00:11 pm, by admin Email , 517 words   English (US)
Categories: IT, Python

Are we on the downhill side yet?

Ryan Tomayko hits one out of the park with his post, Motherhood and Apple Pie. It is the best summary I have read of the state of affairs in software development today, and the competing directions which Sanity and the Vendors are taking.

I found myself thinking, however, that I've been programming professionally now for, what, six years? And it was only with my rather recent move to Python that I really started to dig into "protocols and formats such as HTTP, URIs, MIME, HTML, and even XML (sometimes), and architectures such as REST", or fully understand (and use!) "MVC, ORM, frameworks, test- and domain-driven development". And, not to toot my own horn in any way, but I'm a pretty smart guy; what I mean is, I'm not your average programmer. My guess is that the "average programmers", and their managers, follow the vendors rather blindly because:

  1. they haven't been introduced to some of these more abstract concepts yet, either because they're inexperienced, or are operating on too small a scale, or
  2. they're not smart enough to "get it" when they are introduced to such things.

Even the best programmers spend some time in category 1, as they take time to learn each new concept. I certainly have, and continue to do so. But my hunch is that we've seen a large number of professional programmers in category 2 due to the dot-com glut. Programming, and especially software design, takes a certain set of traits: a good memory, a knack for system organization, the ability to focus, to get into the "flow". But the blitz of advertising in the nineties to "get a high-paying job in computers" has resulted in a majority of programmers who don't have the personality to reach for something better. Quite the opposite, in fact—in my experience, there are some personalities (and lifestyles) that favor having solutions pushed to you, rather than being researched and selected ("pulled") by a more informed process. It's a mistake to believe that only the latter involves reason and judgment; it's simply a different set of factors steering that judgment. But it seems software design is one of those industries which benefits from more people pulling, and less pushing/being pushed to.

My hope is that we are on the downhill side of that glut: witness the recent slide in Computer Science degrees (and careers). The "throw programmers at the problem/product" strategy worked well during the dot-com boom, but doesn't last during the leaner years. I think we'll start to see a return to Sanity, on average, which will result in a swing back toward better design and tools. IMO the current buzz around LAMP stacks, Ruby on Rails, "less is more", DSL's, etc are indicative of that. But I admit I may be blinded by my own learning process, and am projecting what I learn into "what the industry is learning".

Hmmm.

07/12/05

Permalink 09:32:00 am, by fumanchu Email , 1435 words   English (US)
Categories: General

Where there's smoke...

Seems we had a fire recently near our Tijuana camp; I spoke with Forrest Fowler, our Logistics Coordinator, about the event and some of the pictures he took.

Bob: So, what happened last week?

Forrest: I was doing a routine lead-in with a group about 3:00, when Luis called me as I was getting closer to the camp. And he said, "hey there's a big fire at camp." I could see it on the way in, so I wasn't too worried about it. As I got closer I realized the fire was to the south and to the west of the camp.

B: At that point it wasn't on our property?

F: No, and the wind was blowing it away from the camp. But as I pulled into the camp and got closer and started to set the group up, the wind had changed, and was blowing the fire directly towards our camp—in the south area, not where the groups are camped, but where the staff trailers are, and some of the property owner's personal stuff.

Luis had called the [Tijuana] fire department, and they showed up with a truck and a tanker. They started to work over by Luis' house; his house was the most threatened at the time.

Mexican man using a water jug to fight the fireB: Luis Vargas has a house on, or near, our property?

F: Yeah, it's at the back southeast corner of our property. His house has a big, open field next to it, and the wind was blowing the fire right towards his house. So the fire department had their truck and pumper over there trying to put that out. The fire had also expanded a bit toward the north, close to where all of the staff trailers are.

I don't know where they all came from, but there was probably 30 people from the area—just locals—that had come and were starting to help us put the fire out.

Fighting the fire with water trucks, branches, wet shirts--it was crazyB: What were you doing to try to put the fire out?

F: We were running around with tree branches, to smash it out; we had shovels and rakes. One of the caretakers had taken his shirt off and soaked it in water and was beating it down. It was crazy.

Guardiana, the company that delivers the water for our groups—they had just finished filling up the Camping Pros water container. They had some water left, so they came over and backed up to the property-line fence. The fire wasn't actually on our property, but we were trying to put it out on the neighbors' before it got to ours. He started up the generator, and we were spraying the drinking water out on the fire, trying to get it out.

B: When you say there was a fire "south of camp", that's all open field, right? Not an urban setting.

F: Right. It's open grass field fire.

Fire trucks in camp

B: So, you had the fire under control by that point?

F: After about an hour or two of working in that, jumping around to find the different hotspots, the fire department had a couple more engines come out, and we pretty much decided that everything that we could find was out. We had had one spot that we were having a hard time working on, that was over by one of the property owner's warehouses. He has a bunch of wood stakes he used for holding up the grape vines, which was basically a big giant wood pile, and we had a hard time putting that out. But we worked on that for a long time and finally got *that* out.

The fire department said everything was cool there, so Sergio from Camping Pros fired up his barbecue and cooked hamburgers and hot dogs for all the firemen; there were probably 15 to 20 firemen.

FiremanFiremen in the Camping Pros chow line

We had 3 fire trucks, and 4 pumper trucks, and the Subcomandante of the Tijuana fire department, who's the second-in-charge of all Tijuana, was out there. We got to feed them, and take some pictures in front of their engines. They waved goodbye, and I said thanks to Sergio; we talked about it a little bit, and I took off and was headed for home.

Group photo of Tijuana firemen
[Forrest is in the center of the group shot]

B: And what time was that?

F: About 6:00, 6:30. We spent about an hour hanging out eating with the fire department, so about 6:30 or so we took off to go home. I got about a mile or so down the road, right as you get on the toll road, when Luis called me on the radio and said, "hey, the fire started up again." So, I backed up and turned around, and as I got back the fire department was coming back again; Luis had called them as well. They showed up with one fire truck and 2 pumper trucks.

Warehouse engulfed in flame The warehouse in the back that is owned by the landowner, that had caught on fire. By the time we could all get back, whatever was in there was so flammable the whole thing was totally engulfed.

B: The whole warehouse? And what was in there?

F: Yeah. He had a lot of supplies for the farm—generators and PVC pipe and he had a bulldozer that he used for clearing land.

B: Was that all destroyed? Is the bulldozer out of commission?

F: Yeah. Well, a bulldozer's just a big hunk of steel, basically, so if you could figure out how to rewire everything, I'm sure that...you can't really burn the whole steel structure. But it's definitely cooked. And the whole building collapsed--burned down all around it.

B: How big of a building was it?

F: I'd say it was probably... 20 by 50 [feet]. It was wood-frame with corrugated-tin siding and roof. They worked on that 'til just about 8:30, before they were sure everything was out. As they were waving goodbye to go across the road a couple miles, there was a house on fire across the road. So they had not finished their evening.

B: But you had; you felt confident that everything was out. So you went home at that point?

F: Yes, I went home very stinky, my stuff still smells like smoke, and I was tired. But all went well.

B: Was any of Luis' property damaged?

F: No, luckily, nothing of Amor's—none of our caretaker's property, none of our personal stuff—was affected at all.

B: Was there any point at which you had to do some crowd-control with our groups staying at camp?

F: You know, I was amazed. Just because, I guess I project my curious nature upon others; but I only had one person from one of the groups come over and start poking around and trying to help and stuff. And just because of safety issues and concerns, I know that that would not be smiled upon by the administration. ;) I asked him to leave and he said, "that's fine, no problem, I understand." I thought it could have been much more difficult to get him to leave, but he totally understood. I'm just amazed that nobody came over to see more, because it was a big event.

B: I am surprised that you didn't have some panic.

F: Yeah. It was nice because the main view of the fire was out-of-view from where the campers' tents and stuff were set up. The shipping containers and a couple of large trees really blocked the view from where most of the major fire was happening. So they could see smoke, but they never really came over to investigate, which was very good.

B: Is this the first time we've had a fire out there? Is it a common occurrence?

F: This year it seems there have been a lot more fires than I remember. The last time I remember fires like this was about 10 years ago. I'm not sure whether it's because of the new area where we're camping, or if it had to do with so much rain we got over the winter and everything grew a lot more, and therefore there's a lot more to burn.

But that's the first time...well, we did have one fire experience 10 years ago when we were lighting off fireworks at the old camp.

B: (laughing) I remember that!

F: (laughing) but other than that...

B: Yeah. We don't use fireworks anymore, do we?

F: No, nope. Not anymore. :)

B: Well, I think you covered it all very well. Is there anything you want to add?

F: No, I don't believe so.

B: Okay, thanks a lot!

07/10/05

Permalink 09:12:58 pm, by fumanchu Email , 812 words   English (US)
Categories: General

Black & White

Fred Liddell, Tom Liddell, Josh Van Nortwick, and I made up a new pool game a few weeks ago. It's a lot of fun, and only has one glaring defect! :)

Black and White

Object

Black and White is a pocket game. The "black" player must sink the balls numbered 1 to 7 (the "solids"). The "white" player must sink the balls numbered 9 to 15 (the "stripes"). The first player to sink all of their opponent's balls wins.

Requirements

  • A standard set of balls (1 cue ball and 15 object balls)
  • A pool table where both the head spot and the foot spot have matching markers (typically a cloth circle about an inch in diameter)
  • A timer (the one glaring defect, since it isn't standard pool equipment)
  • At least two pool cues (sticks)

Setup

"White" player:

  1. Place the cue ball on the head spot.
  2. Place the object balls 1 through 7 on the head rail, behind the head spot. They must touch the rail, but do not need to touch each other (and probably shouldn't).

"Black" player:

  1. Place the 8 ball on the foot spot. This is your "cue ball".
  2. Place the object balls 9 through 15 on the foot rail, behind the foot spot. They must touch the rail, but do not need to touch each other (and probably shouldn't).

Table setup for the game "Black and White"

Play

Normal play consists of both players shooting simultaneously from their own cue spot, in a series of "rounds". All shots during the game must take place from a player's own cue-spot marker. Each round proceeds as follows:

  1. Both players set their cue ball (the black player's "cue ball" is the 8-ball) on their own cue-spot marker. Their cue ball must touch the marker. If a player is unable to place their cue ball on the marker (because another ball is sufficiently covering the marker), that player's round must consist of dropping their cue ball from a height onto the offending object ball, thereby dislodging it from covering the marker.
  2. Each player indicates they are ready to shoot by saying, "Ready".
  3. Once both players have declared that they are ready, the timer is started. If a third person can manage the timer, 3 seconds is the normal countdown time. If no third person is present, 5 seconds or another time (agreed upon by both players) may be used. If no timer is available, a third person may indicate the shooting moment as follows: hold a cue stick by the butt with the tip pointing straight up, then allow the tip to fall and audibly strike the table rail.
  4. When the timer sounds, both players may shoot. Players are not required to shoot at the moment the timer sounds; they may delay their shot; however, if their cue ball is dislodged from the cue spot marker, they lose their shot for that round. [Optional: As long as a cue ball comes to rest on a marker, the owner of that ball may shoot again without a timer.]
  5. Repeat steps 1-4, unless a penalty shot occurs (see below).

Penalty Shots

During the game, whenever a player makes one of the following errors, their opponent may take a penalty shot:

  1. Sinks their cue ball in a pocket (a "scratch"),
  2. Strikes their own cue ball before the timer has sounded,
  3. Strikes their cue ball when it is not touching the cue spot marker,
  4. Contacts any ball other than the cue (interference)

To take the penalty shot, the opponent places their cue on their cue spot marker, and shoots normally. There is no timer, and the offending player's cue ball is not placed on the table. Errors committed during penalty shots also result in penalty shots.

If both players receive simultaneous penalties, they cancel each other, and neither player shall take a penalty shot.

Resetting Object Balls

An object ball will be repositioned against its starting rail, by the protecting player, whenever:

  1. It is knocked off the table,
  2. It is sunk by a shot made before the timer has sounded,
  3. It is sunk after a player has interfered with play (by making contact with any ball other than their own cue).

Note that each of the above errors also results in a penalty shot.

"Continuous Penalty" Shots (optional)

During the game, whenever a player knocks their own cue ball off the table, their opponent may take a "continuous penalty" shot.

The opponent places their cue on their cue spot marker, and shoots normally. There is no timer, and the offending player's cue ball is not placed on the table. The opponent may continue to place their ball on the marker and shoot, as long as they sink an object ball on each shot.

If both players receive "continuous penalties" during the same round, they cancel each other, and neither player shall take a penalty shot. If player A makes a "continuous penalty" error and player B makes a "normal penalty" error, player B shall take a normal penalty shot only.

Permalink 01:03:08 pm, by fumanchu Email , 91 words   English (US)
Categories: IT, Python, CherryPy

lesscode.org

Link: http://lesscode.org/

I've really been enjoying Ryan Tomayko's new lesscode.org site. I've been on the simplicity bandwagon for about a year now, which coincides nicely with my Python learning curve ;). Check out lesscode if you're tired of overengineering.

Oh, and check out CherryPy if you're tired of overengineered web frameworks. I've worked pretty hard to make the upcoming 2.1 release as simple as possible, but no simpler.

07/06/05

Permalink 10:25:55 am, by fumanchu Email , 151 words   English (US)
Categories: General

36 hours of fun and sun

Our Covenant Presbyterian high-schoolers finally made it to Mexico! We worked on this house:

landscape of city with our house that isn't really very visible

Okay, okay, here's a better picture...we added on to this existing structure:

picture of existing house

Everyone worked very hard, and we accomplished much more than I thought we would. On the first day, we found that the previous group had poured the slab, and built two walls, so we built the rest:

Brandon sawing wood Girls making tar squares with flowers in their hair Brandon pushing a wall into place while Brent nails and Fred assists

(The flowers in the girl's hair came from the neighbor kids)

Then, it was back to camp for the night. During the day, someone had stolen a bag of marshmallows from our van, so Catherine and Ashley went looking for "spares" from another group at camp. Their triumphant return (plus more campfire pics):

high-five for marshmallows Alex covering his face from the heat of the fire Fun fire flare between Tommy and Brent

Next day, we returned to work more on the roof, and simultaneously wrapped the walls in wire, paper, and chicken wire, to get it ready for stucco:

Tommy moving plywood on the roofBrent walking on the roof Grace and Fred finishing the chicken wire

Then it was time for "official" pictures!

Brent taking a picture from the roof(From right) Alex, Fred, Emily, Ashley and 'extras'

07/05/05

Permalink 03:00:51 pm, by fumanchu Email , 292 words   English (US)
Categories: IT

I don't get PUT versus POST

Once in a while, I'll run across posts like Benjamin Carlyle's on REST topics, where the author advocates minimal use of POST, instead preferring PUT for almost every request that has an enclosed entity.

Hogwash. That works fine for blogs and forums, but for real CRUD apps, POST is perfectly fine for updating a resource:

The fundamental difference between the POST and PUT requests is reflected in the different meaning of the Request-URI. The URI in a POST request identifies the resource that will handle the enclosed entity. That resource might be a data-accepting process, a gateway to some other protocol, or a separate entity that accepts annotations. In contrast, the URI in a PUT request identifies the entity enclosed with the request -- the user agent knows what URI is intended and the server MUST NOT attempt to apply the request to some other resource. If the server desires that the request be applied to a different URI, it MUST send a 301 (Moved Permanently) response; the user agent MAY then make its own decision regarding whether or not to redirect the request.

In other words, a URI handles a POSTed entity, but is or becomes a PUT entity. When I make a CRUD app, most of my URI's are "entities that accept annotations". It is a very rare operation for me to replace entire entities.

Or perhaps, I'm just thick. Maybe, Fielding is asking me to expose each attribute of, say, an Invoice resource as its own subordinate resource, with its own URI? But that way lies madness, IMO.

So, stealing the layout from Dave Megginson:

CRUD HTTP
Create PUT
Retrieve GET
Update POST
Delete DELETE
Permalink 08:24:38 am, by fumanchu Email , 87 words   English (US)
Categories: General

People movers writ large

Fedex apparently moves 5.5 million packages every day. In 2003, UPS moved 12 million per day in the U.S. alone. Such companies have thousands of employees, vehicles, and buildings dedicated to moving Thing X from Place A to Place B.

When will one of them consider me a Thing, and move me from Home (A) to work (B) every day at a lower cost than me and my little Nissan?

July 2005
Sun Mon Tue Wed Thu Fri Sat
 << < Current> >>
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30
31            

Search

The requested Blog doesn't exist any more!

XML Feeds

powered by b2evolution