Categories: Python, Cation, CherryPy, Dejavu, WHELPS, WSGI

Pages: << 1 2 3 4 5 6 7 8 9 10 11 >>


Permalink 11:18:48 pm, by fumanchu Email , 200 words   English (US)
Categories: Python, WSGI

Discarded WSGI wrappers

Last year I threw together a preliminary mod_python handler for WSGI (with much help from PJE, of course). Looks like Steven Armstrong has polished it up a bit. Good show!

Here's another adaptation. Anyone interested in consolidating them so a standard can be placed in wsgiref itself?

I've also been thinking about an ASP handler again. Now that I've moved my own Python ASP apps to using a URL rewriter, one of the two strikes against an ASP handler is gone.

The other strike was that the server doesn't have any facility for telling the application whether it's being run multithreaded or multiprocess. The mod_python wrapper dealt with this (in versions less than 3.1) by checking PythonOption directives for the necessary parameters; I think an ASP wrapper could do the same by requiring all ASP+WSGI apps to stuff that metadata into Application.Contents within a mandatory Global.asa. Ugly, but it would nicely round out the WSGI-server offerings.


Permalink 04:25:31 pm, by fumanchu Email , 370 words   English (US)
Categories: IT, Cation

RESTful re-treat

I've been looking around recently at Python web frameworks (again). But I keep remembering the reasons why I wrote Cation in the first place:

  1. When all of your users access network files using a Windows network password, it's very nice to be able to use the same password for a web application—this means using IIS on the server side in its "Integrated Windows authentication" mode. Using IIS means rejecting the vast majority of all existing web frameworks.
  2. A persistent server process is a Good Thing, so "dumb" CGI is right out.
  3. Pure ISAPI handlers for IIS generally suck, due mostly to requiring the deployer(!) to compile a DLL, or costs which I-as-framework-developer don't wish to foist upon others.
  4. ...which leaves ASP. Fortunately, Python can be used as an ASP language. Unfortunately, IIS/ASP has no built-in URL rewriting, so you need a separate .asp file for each URL you expect to serve.

I've been re-examining that set of assumptions lately, because I want to develop more RESTful representations of our data. Many of my current web forms suck, because they pull large sets of data in the interest of not-reloading-the-page. The prevalence of XMLHttpRequest is leading me to investigate alternatives.

The problem with a more-RESTful approach is that I'm currently stuck with the side-effects of #4, above; each URL needs a separate .asp file. So, if I want to publish the URL /app/directory/45987, I've got to create a file named /app/directory/45987! This is Not Good. In addition, I don't think I can serve a resource without a file extension (like ".asp"), but don't quote me on that one.

Therefore, it looks like the choices are:

  1. Ditch IIS and work solely with Apache. This may make some users mad when they get passwords out of sync. But it would mean I could drop my web framework completely and develop a REST+JSON plugin for an existing, major Python framework.

  2. Bite the bullet and use ISAPI rewriting. I found a simple ISAPI rewriter today: ISAPI_Rewrite, which has a freeware "Lite" version.

Tough choices like this make me pine for my own personal hypnotoad.


Permalink 10:48:30 am, by fumanchu Email , 376 words   English (US)
Categories: IT, Dejavu

Outgrowing databases

Daniel H. Steinberg gives a summary of Adam Bosworth's recent keynote at the MySQL User's Conference 2005:

If you build an open source stack that delivers globally available information, how do you massively distribute it and cause it to scale? Bosworth said you need to limit your queries to those that can be easily implemented by everybody and those that can be handled by a single machine. This requires that your queries run at the item level. This might feel odd to those used to dealing with databases, as this means you are not likely to perform joins, aggregations, or subqueries. There is plenty of SQL that cannot be supported.

This is one of the design artifacts of Dejavu: if your domain model requires complicated joins, unions, or subqueries, it's better to refactor your model than to fight with data aggregation queries. Dejavu forces you to do so, in fact, because I didn't care to provide an object-to-SQL translation for such queries.

Refactoring may be painful, but is necessary for growth in almost any application. Best to do it up front than to be lured into fragile schemas, only to be forced to refactor or die later on.

Working backwards through the article:

Bosworth predicts that RSS 2.0 and Atom will be the lingua franca that will be used to consume all data from everywhere.

I've been thinking lately about writing a generic RSS interface for Dejavu. A simple object-property reader/writer would be a cake-walk; security would be the tough bit to design.

Imagine if you can query any data that is available anywhere in the world. Bosworth said that what this requires is a single, simple, open wire format for items. The format needs to be simple for any P programmer to deliver and any JavaScript programmer to consume.

Hm! Exactly where I've been going with Lyrica—I'm working hard to push as much as possible into Javascript on the client. The viewer will certainly be Javascript-only, with an option to traverse local files, so that the server connection isn't necessary once you've built your slideshow.

Permalink 10:09:39 am, by fumanchu Email , 286 words   English (US)
Categories: IT, Python, Dejavu, Cation

Designing from the outside in


...extracting a re-usable framework after the fact struck me as interesting because that's really what's happened with Leonardo. Two years ago, I wrote a little wiki-like script in Python in order to enable editing of content on from a browser. I then decided to expand it just over a year ago to include a blog. Now, as more features are being requested, an underlying web framework is emerging that could very well be useful outside of running a wiki or blog.

The same thing happened with Cation (a web framework) and Dejavu (an ORM). I was tasked with rewriting our core business app—two years ago, it was a procedural CGI app written in Visual Basic 4! When I rewrote the whole thing in Python, I started by isolating Cation+Dejavu into their own layer. After about six months I then separated Dejavu from Cation. In addition, I made a middle business-objects layer called "EnDue", which the final app, "Mission Control" is built on. There's also a wiki-like app called "Junct" which I built on top of Cation and Dejavu. So the tree currently looks like this:

[Cation]  [Dejavu]
    \        /
     \      /
[Mission Control]

I've also got "test apps" for Dejavu and EnDue (well, I'm still writing the one for EnDue...I think I'll model the business of the beard-and-stone salesman from "Life of Brian" ;) Any good names for such a business?).

Anyway, the real point I want to make (and have made before) is that I'll probably replace Cation with another web framework sometime this year...but I wouldn't have known which existing framework to pick if I hadn't written my own first.


Permalink 03:56:35 pm, by fumanchu Email , 368 words   English (US)
Categories: IT, Dejavu

Trigger unhappy

Bah. I can't design my way out of a wet paper bag when I get rushed, and I've been awfully rushed the last month or so.

The problem is update triggers. Our business (like every other) has tons of things that should happen when data changes. For example, when one of our clients decides to arrive a day earlier, that affects lots of business decisions. The people who make those decisions need to be notified, usually by email. Other, dependent business objects need to have their data updated. The very act of changing the date needs to be logged.

OK, that's not the real problem yet. The real problem is finding out when to do all of these activities. My naive first approach was to simply perform all of these side-effects whenever the FirstDate property of a MissionTrip unit changes:

def __set__(self, unit, value):
    if value != unit._properties[self.key]:
        unit._properties[self.key] = value
        self.fire_triggers(unit, value)

This is straightforward, but not very flexible. I put myself in a corner recently over this: the FirstDate is calculated based on a set of TripDate objects. When a user submits the MissionTrip web page, they might add three new TripDate objects—each one triggers a recalculation of the FirstDate property, and I ended up with three separate records in the log, along with three separate email notifications!

The local fix would be to not put a trigger on the TripDate objects, and to manually recalc the FirstDate. Wherever needed. There goes the whole point of objects.

What would be better would be a trigger that could fire at the end of the session (say, the web-page submit) instead of at the point of modification. But I can't figure out a clean way to do that. I worry that I won't remember to call session.cascadeAllTriggers() at the end of some submit handler, and I won't notice that fact for some time, since all of those activities are by nature side-effects. Not to mention the corner cases where I need to fire triggers before the end of the session.

Hmmm. I'd like something more declarative than imperative, I think.

What a messy part of domain modeling. :(


Permalink 11:31:22 am, by fumanchu Email , 750 words   English (US)
Categories: IT, Python

The hymn microformat (HMML?)

Dave Warnock has been talking about a new microformat for hymns, and has chosen an initial set of primitives that are surprisingly close to what I've had in Lyrica for some time now. I wrote the first version of Lyrica a couple of years ago, and the whole application seems rather baroque to me now, for many different reasons:

  1. The tools to put together a slideshow are IE-only, because
  2. The app runs serverless, which means it needs an ActiveX object to write local files.
  3. The make-a-slideshow UI sucks in many ways. I don't like the load/save flow in particular.
  4. Although the song files themselves are quite minimal, there's too much layout information contained in them—the order of verse-chorus-etc shouldn't be hard-coded by the order it appears in the song file. I mistakenly told David that I never repeat choruses—turns out I have on a few occasions to get around the limitations of my design.

There are, however, several design choices I still like:

  1. The song files are pretty minimal, and approach Dave's concept of an open format for hymns. If you're looking for a name for it, Dave, try HMML: Hymn Microformat Markup Language. "Himmel" is German for "Heaven" ;) (among other meanings).
  2. The javascript for running the slideshow is nice and simple, and extensible with custom plugins per style.
  3. The CSS is simple, partly because of the microformat.

Dave talked about using Eric Meyer's S5 for the transform from bare content to slideshows. I still don't like S5's one-style-per-show design—Lyrica allows my church to apply different styles to each song pretty quickly, without hand-editing any files. So I think I'll revamp it again here Real Soon Now. I'd like to see HMML become transparent enough that both S5 and Lyrica operate on it with little fuss, but I simply don't see S5 having a model which fits Sunday morning song lyrics. Sermons, maybe, but not songs.

What I'll focus on in a redesign:

  1. IE-only has to go, which means a server will have to handle the song-editing and slideshow-composition persistence. I resisted requiring a server for a while, since churches tend to lag behind the tech curve. However, if a stable microformat can be reached, I can probably get the server component down to a very small standalone Python webserver, which can run on localhost:8080 for example. With an appropriate, minimal REST spec, the server side could be written in a variety of languages.
  2. Most of the slideshow-composition tools will get some nice AJAX flows. No more modal dialogs. Hooray!
  3. I'll look at moving the lyric flow into Javascript instead of the current 'slide' divs.

No, better yet, I'll move the lyric flow out to the operator. Hymns may be sung the same way every time in every church, but modern praise music is certainly not. One church might sing Verse-Chorus-Verse-Chorus while another might sing VVCV-Bridge-CC-Tag. Many praise bands replay lyric blocks "as the Spirit leads", both interminably and randomly, it would seem. The slideshow operator, therefore, needs both a "standard" flow template and tools to modify that flow on the fly.

I strongly believe, therefore, that the microformat should neither repeat lyric blocks, nor should it contain flow markers inline like Dave's "repeatchorus" paragraphs. The flow of blocks should be pushed out to metadata at worst, and out to the operator at best. A metdata solution might be a meta name='default_flow' content='verse1 chorus verse2 chorus' tag or something similar. An operator tool might be a set of JS key bindings available while a given song is displayed. Perhaps a nav overlay could be displayed while the layout decision is being made? Arrow-keys could follow the default flow specified in the meta tag.

Finally, a microformat that moved flow out of the main content would cut out an enormous amount of wasted duplication. I already have some songs with multiple files, because the words are slightly different between versions. A central repository of songs would probably see an explosion of versions, resulting primarily from differing lyric arrangement.

A flow-free microformat would also make writing a HMML editor much easier.

Hmmm... much to think about.


Permalink 08:49:45 pm, by fumanchu Email , 171 words   English (US)
Categories: Dejavu

Bug in

There was a bug in until today (revision 69). You should probably upgrade from SVN if you're using dejavu.

The bug manifests whenever you have a logic.Expression that has a function call which cannot be represented in SQL. For example, I hit it with the Expression: lambda x, **kw: x.FirstDate <['Year'], 12, 1).

There are two parts to explaining why this went unnoticed for so long. First, all of my uses of had been constants so far. When they are made into Expressions, the early binder turns the whole .date constructor call into a LOAD_CONST. Consequently, I've never written a dispatcher for the constructor, which is a function. So db.SQLDecompiler.visit_CALL_FUNCTION had no concrete function object to call, and nowhere to dispatch to. When both of those most-common cases fell through, the decompiler stack should have gotten a cannot_represent entry appended to it; instead, that entry was overwriting the top-of-stack. Ugly and oh-so-fun to track down.


Permalink 10:23:55 am, by fumanchu Email , 69 words   English (US)
Categories: Python

Got my 2nd edition Python Cookbook, today

Had to leaf through 757 pages to find my name ;) but it's there. Now if I can just figure out what I contributed to that recipe...

Ah. Here's my comment on Stephen's excellent recipe. Hm. I still don't see how any of my comment got into the printed recipe. Oh, well. I'm just happy to have a complimentary copy of the Cookbook. ;)


Permalink 12:06:16 pm, by fumanchu Email , 864 words   English (US)
Categories: IT, Python

Linguistic interfaces may be the future, but not for programming

Metafor is a new attempt to bring a linguistic user interface to programming.

Where to begin in describing how wrong-headed this is?

You've got the wrong target, guys. Programmers do not need a higher level of interface for their code. It's been tried with graphical environments for decades (and has mostly failed); most of the good coders I know avoid graphical IDE's, preferring plain text every time. What makes you think an even higher level of input will work better? Typing plain text is a developmental optimum for readable, maintainable code. We don't need a code generator to produce boilerplate; we need languages which don't need boilerplate, like Python.

But wait, you say, we're not targetting programmers, but the "general public". Okay, you've got the right audience for LUI's (Linguistic User Interfaces), but the wrong task. "Natural language is so semantically rich and flexible that if it could be computationalized as a programming language, maybe everyone could write programs". At one level, yes programming is design and everyone designs things. But programming of the sort I do is so far beyond the reach of the general public that a new interface means nothing. The gap you're apparently trying to bridge is between user's mental models and a running computational system, which I think you think is created by the steep learning curve of formal languages. Way off base. The gap is rather between users' incomplete, fuzzy, situational model and complete, strict, formal systems. That gap takes decades to learn how to bridge, by human brains who are constantly trained to do so.

In the researchers' study of seven intermediate programmers and six beginning programmers, subjects who saw their sentences translated into code this way were "seemingly quite surprised that all that information was contained in their utterance[s]," said Liu.

Funny how you didn't quote any advanced programmers. Developers who have been around the block know that the prototype you might generate from initial end-user conversations is going to be:

  1. Wrong. Not just mildly wrong in the details, but so divergent from a practical model of reality that a human is needed to understand the humans from a human standpoint, let alone from a machine point-of-view. Humans don't know what they want and cannot explain it when they do know what they want.

  2. Wronger. The details will be wrong, too. Humans love to have 38 names for the same thing; computers hate that. Humans love to have special cases; formal systems really hate that. Humans love to change their minds, not just about content, but about deep structure. They do it fluidly and relatively effortlessly. Humans love to do (and evolve) things first, and describe and analyze it later. Programs suck at that.

  3. Wrongest. Everything about the program will change in six months. The users will decide they don't really want Pacman to eat the dot in the upper-right-hand-corner on alternate Tuesdays, if Blinky is in the spawn box but all of the other ghosts are out. No, scratch that. One user wants that, all of the others mean every third Sunday. No, scratch that. Half of them don't know which one is Blinky. Just don't show Blinky to them. No, wait--it's not enough to not display Blinky; if you turn off the display but not the effects of Blinky, your users will come after you with pitchforks and torches. Show me one non-programmer who understands these implications and can convey them to a natural language visualizer (in a sufficiently declarative conversation) to produce a computer program.

Seeing language translated to logical code on-the-fly made the users more conscious of their communicative precision, said Liu. The subjects' quickly began using language that was simple and declarative, which, in turn, improved the usefulness of the system for brainstorming and outlining, he said.

As one of our customers said the other day, "double duh!" You selected programmers to test the value of your system to the general public? That's so misguided it's not even wrong. "Using language that was simple and declarative"? That's what I do all day; that's what programming is. That's the learning curve you're trying to avoid. Now that you've failed to avoid it, you've done nothing other than introduce a serial (as opposed to parallel) development process in-between your developers' mental models and the code. Have fun with that additional overhead.

Wild tangent: nobody, not even programmers, wants to type (or say) "the bartender gives the customer a drink only if the drink is on the list of drinks on the menu." They want to say, "he gives it to him only if it's on the menu." Give it pronoun awareness, and I'll re-review it.

So what's the "right" target for something like this? Linguistic interfaces are nice things; I don't dispute that. And they're appropriate for the general public. But they're not usable for the design of formal systems yet. I'm ready for a great linguistic interface to my domain-specific application, with which my users can manipulate a model of a subset of reality. I don't know of anyone who wants to use an LUI for building the app itself.


Permalink 11:44:17 am, by fumanchu Email , 327 words   English (US)
Categories: IT, Python

Lightweight services need a lightweight server

I've been meaning lately to take my web presentation tool, Lyrica, and rewrite it. It's currently IE-only because it's a local page, and needs IE's ActiveX to write local files.

What I realized today (reading Simon Willison describing Greasemonkey to Jon Udell), is that I tend to approach web servers as fairly heavyweight things. You've got to set up a webserver, plug in a CGI language like Python, deal with app-specific state somehow, run a DB, etc. etc.

What I'd rather do, and what RESTful design + Ajax allows me to do, is design a local page which requests remote objects. A lot of end-user apps in this space tend to be heavyweight because they serve the HTML interface page and the data objects (used by that page) from the same server. I'm now envisioning a very lightweight, standalone server for the data, but making the interface a local HTML page—that interface HTML page will use XMLHttpRequests to persistently manipulate the user objects.

I like Simon's term "lightweight intermediary". If the Web server can be viewed as a lightweight intermediary between the local (i.e. not served from a web server) Ajax page and the data persistence layer, the design of that intermediary can become much more abstract and lightweight; that is, the web application can be broken into an abstract persistence mechanism that is very small, while the heavy interface design can be separated into a local page which is NOT served by the web server.

This design, then, really hits on the concept of programmable Web services, enabling different developers to design their own interfaces on top of the isolated data layer. Cool. I can hopefully then focus on a very nice data format, and let evolutionary development take its course with that, and reach more users.

I'll let you all know how this goes with Lyrica--it should be straightforward.

<< 1 2 3 4 5 6 7 8 9 10 11 >>

March 2017
Sun Mon Tue Wed Thu Fri Sat
 << <   > >>
      1 2 3 4
5 6 7 8 9 10 11
12 13 14 15 16 17 18
19 20 21 22 23 24 25
26 27 28 29 30 31  


The requested Blog doesn't exist any more!

XML Feeds

powered by b2evolution