Mass Transit

We've (hopefully) all seen C# 3.0's fantastic new feature called LINQ. I think it's a great idea. It makes it easy to query differing data sources using an embedded query syntax. Wouldn't it be great if we had something like this for Python?

We do. It's called a list (or generator) comprehension. For example, it lets you rewrite this LINQ example like this:

numbers = [5, 4, 1, 3, 9, 8, 6, 7, 2, 0]
lowNums = [number for number in numbers if number < 5]

The problem is, we can only run these comprehensions inside the Python application, which is inefficient if you want to talk to a SQL RDBMS with a huge amount of data to filter.

Enter SQLComp. SQLComp breaks down the ASTs and translates your list and generator comprehensions to SQL, and automatically queries the database and gets the result. It also allows simple, safe variable interpolation. The wiki page has a few simple examples on it.

Please note that SQLComp is experimental, and I intended for someone to perhaps pick it up, hack on it, and make something cool. Shoot me an email if you're interested in collaborating.

Why don't you give SQLComp a look?

PyMeld is the coolest templating language ever, because it's not a templating language, it's a template engine. It's what I like to call an "active" or "push" templating engine, that is, rather than the Cheetah school of passing a model object or dictionary to the template and letting the templating engine "pull" the data from it and figure it out, your controller code handles the vast majority of the logic ("pushes" the data), and the engine just makes it simple to manipulate the document. This is the best way for a templating language to be, because your templates are similar, and you're using a Real Programming Language to do what Real Programming Languages should be doing instead of these fake templating languages. In addition, your template files are nice and simple, and can be pulled right out of Dreamweaver and your graphic designer without having to mark it up with stuff.

PyMeld isn't without its issues, though. It's really slow and inefficient, as its powered by regular expressions and must reparse every time you run an operation on it. It's license requires you to pay money to use it commercially, which sort of sucks too.

There's a few reimplementations around, but they're very ZPT-like and, I believe, miss the point of PyMeld. PyMeld is just a badass, amazing, extra-special DOM library, and these reimplementations try to turn it into YAPTL. Meld4 sticks to PyMeld's roots, but throws the power of ElementTree behind it. I also added a couple of utility functions: fill() implements something similar but cooler than ASP.NET's MasterPages, and push_pull() is for those people who really can't bear to use an active templating engine, and will accept a dictionary that will fill the keys of a template and automatically handle lists, nested dicts and primitives.

Why don't you give Meld4 a look?

Recently, it seems like everyone's been focused upon creating The One Framework -- the Ruby on Rails of Python. In fact, I'm almost as guilty as everyone else. We have failed in this regard, and we will continue to fail. We've been struggling to find the best practices and combination of tools to create Joe User's average database-backed dynamic web site and application, and not only can't we agree on how to do it, no actual end-users care when polished solutions like Rails and ASP.NET are in town. Not only that, by many accounts Ruby seems to be overtaking Python in its rate of growth. Personally, I'd rather write Python.

But oh, how quick are we to forget Python's smashing successes in the Web world. We've got Plone, probably the most popular and comprehensive CMS out there. We've got the platform its built on, Zope, which is (from what I hear) fairly popular, too. We've had PyBloxsom, which until WordPress came along was a very widespread blogging system. Let's also not forget MoinMoin and Mailman, some of the most widespread Wiki and mailing list applications on the Web today. And hell, we've got Python's "killer" web app, Trac, which just about everyone is using these days. And oh, BitTorrent. I know it's not a Web application in the traditional server-side sense, but it just shows how capable Python truly is.

What do all of these applications have in common? They certainly don't share a web framework; most of them are developed specific to that application and tailored to various deployment platforms (CGI, FastCGI, custom server, mod_python). Python appears to be falling behind in the world of custom-made Web applications, but in the world of generic, reusable Web applications, Python is doing great.

And let's not forget about the great achievement of Web-SIG: WSGI. Now, we have a system that lets us write an application once and deploy it upon any WSGI-compliant server or gateway (to name a few, ISAPI, mod_python, CGI, FastCGI, SCGI, custom HTTP server).

Here's what I propose: screw Web frameworks for now. We won't win in the Rails generation. Perhaps when some ingenious Python programming comes up with that next-generation Web framework (Seaside + ZODB + LINQ + ASP.NET + PyMeld + LivePage + CrackAJAX, anyone?), we can give the Web framework wars another shot.

For today, let's work on making generic Python products. Let's make a kick-ass community forums system, an incredible blogging system, a news script, a CPanel/Webmin clone that people will use because of their features, not their programming language. Commentary is a great example of what I'm talking about. Let's not forget to make sure Trac, Plone, and all of our applications run seamlessly on WSGI, and let's make our WSGI gateways ironclad and diverse.

And, most importantly, let's all get behind what I believe is the most pressing and important concept in the Python web development world today: Paste Deploy. What Python needs more than anything is a brainless, quick-and-easy way of connecting applications to gateways. I want to drop a CGI file on to a Web server, point its config file to (the fictional) pyBB-1.0.0.egg, chmod it, and have it Just Work. I want to do the same thing for FastCGI. In fact, what I would love more than anything would be a portable mod_wsgi across Apache, LightTPD, and IIS: a module that would let me drop a .egg file into a directory and have it automatically pick up and install the WSGI application from the archive. Once we've got this, a standard, portable way of easily installing ANY Python web app, we'll be getting somewhere.

Happy hacking.

Link: http://www.gosubway.org/

Subway 0.2-rc1 is out. http://www.gosubway.org/

http://svn.subway.python-hosting.com/crackajax/trunk

For some reason, itunes.py doesn't have remote_eval functionality in Opera. It used to work, I swear...I just broke it...

As many of the #cherrypy regulars know, I've been developing a top-secret Ajax framework code-named CrackAJAX, which will soon be integrated into Subway. Though it's not quite ready for a release, I will show you a quick teaser of its current functionality, and some of the motivation behind developing it.

I hate Ajax. It's a silly name, and it's bad technology. JavaScript is a horrid language to develop in, and the destandardized DOM that exists in the current browser market makes maintenance a nightmare. HTTP wasn't designed to support this sort of thing, and Ajax breaks URLs and the back button.

But it's a cool idea, and the functionality is great.

Well, there wasn't much I could do to fix the latter problems, but I figured, what if I could write an Ajax application in Python? I set out to work, and came up with CrackAJAX, which uses the jsolait JavaScript library.

CrackAJAX is based around the concept of an AjaxPage, an HTML page that is "ajaxified." In its current incarnation, you can write the HTML code right in the docstring, or pass it a callable template rendering method. You extend the AjaxPage class, add your markup, and then add special methods. These special methods are decorated as either @clientside or @serverside, depending on where you want them to be run. @clientside methods translate the Python source code of the method to client-side JavaScript, and @serverside methods automatically generate an XML-RPC call to the server. You can call these seamlessly from your DHTML handlers, just like regular JavaScript.

It's quite a joy to develop with, actually. I built an Ajax application that emulates iTunes' search functionality. Granted, the JavaScript loops are a bit dicey, but look how slick this is:

import crackajax
import cherrypy
import ituneslib

class iTunesAjaxPage(crackajax.AjaxPage):
    """
    <html>
    <head>
    $crackajax
    <title>CrackAJAX iTunes Example</title>
    </head>
    <body onload="load_list()">
    Welcome to the CrackAJAX iTunes example. This search system is similar to
    the iTunes search mechanism; just start typing characters to limit your search.
    This application is pulling data directly from a "Library.xml" file located
    in the current working directory.
    <form>
    <b>Search:</b> <input type="text" id="searchbox" onkeyup="do_search()" />
    </form>
    <ul id="songlist">
    </ul>
    </body>
    </html>
    """
    def __init__(self, lib, *args, **kwargs):
        crackajax.AjaxPage.__init__(self, *args, **kwargs)
        self.SONGS = []
        lib.load()
        try:
            while True:
                track_info = lib.getTrack()
                if track_info:
                    self.SONGS.append(track_info.get("Artist", "(no artist)") + " - " + track_info.get("Name", "(no name)"))
        except EOFError:
            pass

    @crackajax.clientside
    def load_list():
        update_list(get_all_songs())

    @crackajax.clientside
    def update_list(songs):
        the_song_list = document.getElementById("songlist")
        while true:
            if the_song_list.childNodes.length == 0:
                break
            the_song_list.removeChild(the_song_list.childNodes[0])
        c = songs.length
        if c > 0:
            i = 0
            while true:
                song = songs[i]
                newitem = document.createElement("li")
                newitem.appendChild(document.createTextNode(song))
                the_song_list.appendChild(newitem)
                i = i + 1
                if i >= c:
                    break

    @crackajax.clientside
    def do_search():
        update_list(search_songs(document.getElementById("searchbox").value))

    @crackajax.serverside
    def get_all_songs(self):
        return [self.SONGS]

    @crackajax.serverside
    def search_songs(self, query):
        query = query.lower()
        return [filter(lambda s: self.does_song_match(query, s), self.SONGS)]

    def does_song_match(self, query, song):
        song = song.lower()
        words = query.split(" ")
        for word in words:
            if word not in song:
                return False
        return True

crackajax.init("jsolait")

cherrypy.root = iTunesAjaxPage(ituneslib.Library("Library.xml"), "")

cherrypy.server.start()

Over at the Subway project, we've been getting ready for the 0.2 release. This release is a huge stepping stone to Subway 1.0.

Subway, a Python "megaframework", was designed from the ground up to ensure that every single line of code the developer writes is meaningful. It was also designed to keep the total lines of code down without sacrificing readability and ease of maintenance. With so many Web frameworks in the Python world, why did I write Subway? None of the current Web frameworks included everything needed to create a Web application while still remaining flexible (I'm looking at you, Zope). In doing so, I realized that there were still a lot of great tools out there. I decided to reuse as many existing Python components as possible, and as a result, we have a framework that has more mature component parts, a framework that is not the quintessential reinvention of a generic web framework, and a little bit of good PR.

A lot of people have wondered how it compares to other frameworks. Granted, there is a lot of overlap between the various projects, and I'm going to try to explain how Subway is different.

The first (and most controversial) difference that people usually notice is the choice of a templating language, Cheetah. I had several reasons for eliminating contenders such as Kid, ZPT, PTL, DTML, Nevow, PyMeld, my own OneRing and PSP in favor of Cheetah. First of all, Cheetah is already known as one of the most mature templating systems around. Although it is a version 0.9 product, it has many years of development time behind it. Furthermore, such maturity brings mature features, features like object-oriented-document page inheritance to manage larger websites, and excellent documentation. Cheetah is also known for its speed due to its excellent precompilation mechanism. I also wanted a templating language where users did not have to go through a paradigm shift; they could go from hacking code in PHP or standard Python right into Cheetah with a minimal learning curve. Finally, templating languages like Kid, ZPT, Nevow and PyMeld are HTML-only, meaning they cannot generate other document formats like CSS. In my mind, this is unacceptible. Some of these language require your documents to be XML-compliant, which is another no-no. A framework (and templating language) should never get in your way, ever.

Though TurboGears in particular is very, very similar in structure, Subway has a lot of lesser-known features. One of the cooler ones is CherryFlow, a generator-based system that allows you to write a complex Web application as a flow of different pages all in one method. It looks something like this:

@expose
@flow
def example_flow():
    yield view.first_page()
    if request.args["choice"] == "a":
        yield view.choice_a_page()
    else:
        yield view.choice_b_page()
    yield view.last_page()

This sort of functionality is analagous to Cocoon's FlowScript.

Subway also has a form validation framework that makes your life easier. In my incredibly biased opinion, it's the best form validation out there. Every Subway "view" (template object) has an attribute "form", which corresponds to a FormEncode schema that is generated by reading specialized tags and attributes inside the view file. These definitions look something like this:

<form method="post" action="submit_form">
Your name:
<input type="text" name="yourname" form:required="true"
    form:message="You must enter your name" />
<form:error name="yourname" />
<br />
Your e-mail address: <input type="text" name="youremail"
    form:validate="email" form:message="Your must enter a valid e-mail address" />
<form:error name="youremail" />
<br />
<input type="submit" value="Submit" />
</form>

Next, you'll want to hook this up to some Python code to handle the form. That code will look something like this:

@expose
def display_form():
    return view.the_form()

@expose
@form_handler(view.the_form.form)
def submit_form(yourname, youremail):
    if form_submitted:
        if form_errors:
            return view.the_form() # send back the form, Subway will automatically fill in the fields with their submitted values and display all error messages
        else:
          # form is OK
            return view.form_ok()
    # form was not submitted
    return view.the_form()

You can also manually manipulate the form_defaults and form_errors dictionaries to add your own custom validation, or provide a custom-defined FormEncode schema to the form_handler decorator.

A handy little feature of Subway is the docstring views. An example shows it clearly:

@expose
def test_docstring_view():
    """
    This $name is in a docstring
    """
    return test_docstring_view.view(name="view")

Before I sign off for today, let me remind you that a super cool new Ajax framework will be included with the next Subway version, 0.3. Keep an eye out.