Categories: IT, Architecture, Linnaeus Award, Python, Cation, CherryPy, Dejavu, WHELPS, WSGI, Robotics and Engineering

Pages: << 1 ... 5 6 7 8 9 10 11 12 13 14 15 ... 17 >>


Permalink 12:37:37 am, by fumanchu Email , 159 words   English (US)
Categories: General, CherryPy

A severe shortfall in the motivational index

I can't seem to finish or maintain anything lately. I never can, but recently it's been more acute. Maybe it has to do with the pending release of CherryPy, which means we're in "feature freeze" mode; I miss writing the sweeping redesigns that I was able to shove into 2.1 before the beta.

But it's apparent in other areas, as well. I have about 80 hours of video transfer to DVD to do for various friends (and some personal projects). I haven't touched any of them in months. I also have a company to officially close, which is moving at a glacial pace. I've got a bunch of papers in triplicate that I need to mail off to the State, but can't seem to work up the drive to buy a large envelope for the purpose.

But then, having everything "done" is a much worse state. Maybe I should count my busy blessings.


Permalink 03:48:36 pm, by fumanchu Email , 232 words   English (US)
Categories: Dejavu

Dejavu has a new home

Dejavu, my pure-Python ORM, has a new Trac home at As always, it's open to the public to download, use, or develop. It could use another contributor or two, and the snazzy new Trac front-end should help that immensely.

Just to remind you about the parts of Dejavu I think are cool:

  1. It completely hides the storage layer, so your code need never know you're using a database. It supports MS Access, MS SQL Server, PostgreSQL, MySQL, SQLite, and shelve, all transparently.
  2. Simple queries use simple syntax: Panda = sandbox.unit(zoo.Animal, Species='Ailurpoda', Name='Mei'). More powerful queries use the logic.Expression object and pure Python (no code strings!):
    red_filter = logic.Expression(lambda x: x.color == 'red')
    RedAnimals = sandbox.recall(zoo.Animal, red_filter)
  3. Dejavu has been designed from the ground up to be used in a multithreaded environment.
  4. New database adapters are easy to develop using the `db` module. The PostgreSQL module, for example, is 100 lines of code.
  5. Model-layer Units are clean and easy, using modern, built-in Python types. They are also extendable to custom property classes and custom types:
    from dejavu import Unit, UnitProperty
    class MissionTrip(dejavu.Unit):
        """A Mission Trip experience."""
        DirectoryID = TripGroupProperty(int, index=True)
        FirstDate = NotifyProperty(
        GroupName = UnitProperty()
        LastDate = NotifyProperty(
        AmountDue = UnitProperty(fixedpoint.FixedPoint)
  6. Persistent query-engines and analysis tools are included.


Permalink 05:52:07 pm, by fumanchu Email , 359 words   English (US)
Categories: CherryPy

Eat less, exercise more

Dave Warnock muses:

I am not about to argue that [TurboGears and Subway] should merge, instead I feel they can improve most by making sure that they each stay thin putting all the improvements they can back into the components eg CherryPy (which they both have been doing) and into the deployment elements (setuptools and paste).

That's a good point, and I think it's at the heart of what CherryPy is trying to be: non-fattening. So there's definitely going to be some pushback from CherryPy itself trying to "stay thin".

I often say that CherryPy is not a "web framework"; it is an "HTTP framework". That is, it doesn't try to provide tools for every facet of web development. Instead, it concentrates on wrapping HTTP up in a Pythonic way.

IMO, working to stay thin is an important factor in getting CherryPy "more exercise": it gets used in more meta/mega-frameworks like Subway and TurboGears precisely because it hasn't gobbled up every good idea, just because it's web-related, or even just because it's Python. For example, CherryPy 2.1 is deprecating the Aspect module that was in 2.0, because it isn't related to the HTTP-focus of CherryPy.

David goes on:

Another project that is the next level down from these frameworks but that is also moving fast is Quixote, I feel the differences between Quixote and CherryPy are also becoming smaller (shown by the recent blog posts on Python Web Controllers). Whether they could ever merge is a different matter. Probably not possible (or even desireable) for the moment.

I would have to agree. There's a decisive difference in architectural style between CherryPy and Quixote. That doesn't mean there aren't components that are common to each, and there are certainly some which are unique to each which deserve to be ported! If the Quixote coders are willing to give up all the method names starting with set_ and get_ we're ready to have a conversation about merging. ;)


Permalink 04:09:43 pm, by admin Email , 65 words   English (US)
Categories: CherryPy

The medusa called autoreload

This is what I spent my weekend working on (among other things). It's the "autoreload" functionality in CherryPy. It was so complicated that it took me 15 minutes to understand it again, anytime I got distracted; having the diagram makes it quicker, at least. They say the human brain can handle about 7 things simultaneously, and this snake-pit takes about 4 at a minimum:

Permalink 08:50:13 am, by admin Email , 54 words   English (US)
Categories: IT

Fractal frustration


Okay, coworkers, now you know: when I put my hands over my eyes and sigh heavily, it's because I'm on step 8. Actually, the more I learn about IT, the better I get at seeing step 8 coming when you're still on step 2. So please bear with me whenever I seem unduly frustrated at small requests.


Permalink 11:52:49 am, by fumanchu Email , 193 words   English (US)
Categories: CherryPy

CherryPy 2.1 RC1 is out

The official mailing-list announcement is here.

The big change from 2.1 beta is the session filter—it's been completely rewritten.

Minor updates/fixes:

  • Server-side image maps (ISMAP) now supported.
  • More documentation in the official CherryPy book.
  • Improved coverage tool output.
  • Support for partial GET requests.
  • New HTTPError(status) exception, plus pretty HTML pages for 4xx-5xx responses.(which are customizable).
  • Separate access and error logs.
  • % HEX HEX decoding now works for URL's, not just params.
  • New cptools.serveFile function.
  • New config entries which allow you to limit the size of request headers and body (to avoid denial-of-service attacks).
  • Tracebacks can now be inserted into the CherryPy log via "server.logTracebacks" config entry (True by default).
  • New expose() function/decorator, which allows you to alias any page handler method.
  • HTTPRedirect can now be raised in _cpOnError or error filter methods.
  • Other minor bugs in the beta were fixed.

Way to go, team. I'm pleased as punch to be a part of this powerful, Pythonic product. :)


Permalink 10:04:45 pm, by fumanchu Email , 119 words   English (US)
Categories: CherryPy

mod_python wrapper for CherryPy

I wrote a WSGI wrapper for modpython a while back, but nobody's gotten it to work yet with Apache2 on Linux (at least, nobody in the CherryPy community). The current theory is that it's due to the differences in MPM's between Windows (mpm_winnt, which is one process with multiple threads) and Linux (worker, threadpool).

If you're stuck wanting CherryPy + modpython on Linux, have a look at Jamie Turner's new, which skips the WSGI layer and directly connects CherryPy to modpython. Then let everyone on #cherrypy know how it went. ;)

Permalink 05:58:38 pm, by fumanchu Email , 507 words   English (US)
Categories: IT, General

Broken mail, and when to outsource it

[Disclaimer: if you're a techie, and you find this discussion a bit patronizing, bear with me. I raise my own salary through donations, and I'm trying to communicate to my non-technical financial supporters where their money is going. If you're one of those supporters, please bear with me when I descend into meaningless jargon. ;)]

Our mail server, which runs Microsoft Exchange, died two weeks ago—critical operating system files got corrupted, and Windows wouldn't boot. This is a bad thing. Bad technically, because it meant reinstalling Windows, which was sure to confuse the Exchange service. Bad corporately, because it took us two days to get mail services back up and running, which meant lots of business not getting done. In point of fact, although we spent several hours with a consultant and a Microsoft engineer, we didn't actually fix the problem; however, we got the mail server back into enough of a working state that business could proceed. So we backed up all our mail and said to ourselves, "don't touch this machine again".

Then I started shopping. Being an intuitive fellow, I listened to that little voice in the back of my head saying, "now would be a good time to outsource our mail". That is, I went looking for someone who could run an Exchange server somewhere else on the Internet for us, and host our mail. If it worked, this would be a big benefit to us because:

  1. It would make someone else squirm when the server crashes,
  2. The provider would benefit from an economy of scale, since they perform the same service for multiple companies: they could afford redundancy and safeguards that we, a small company, cannot,
  3. It would take the administrative burden off Ryan, our system administrator. At the least, it would change his handling of issues away from technical resolutions (fixing machines) to social resolutions (calling the service provider when things go wrong),
  4. At $10/month per mailbox, it's pretty cheap for 40 users.

Would I have made the same decision if we had 20 users? 100 users? Perhaps. What was more important in my calculations was the ratio of IT workers to other staff. We are right at the point where we need more IT staff, but haven't hired them yet. That is, the demand for IT is outpacing what Ryan and I can supply.

Assume an ideal ratio of 1 IT worker for each 20 staff members. Given our staff of 40 (and growing), if we had 3 IT workers, we might have chosen to keep hosting our own mail. Similarly, if we had 20 staff to our two workers, we would have chosen (and did choose for the past few years) to host our own mail. But because our demand is exceeding our supply, we chose to outsource it for now:

Ratio of IT to staff per year

We're still in the process of moving our mail to the new provider, but don't worry about sending us mail! Nothing about that will change. I just wanted to give some of you a peek into the day-to-day (-to-week-to-month?) fires I end up putting out.


Permalink 01:06:38 am, by fumanchu Email , 676 words   English (US)
Categories: CherryPy

CherryPy now handles partial GETs

Partial GET requests are a handy way for a client to request a portion of a resource, rather than the entire resource. HTTP clients send a Range: bytes=start-stop request header, where start and stop are non-negative integers. The HTTP server can then send only those bytes (inclusive) in the response. Multiple byte ranges are also possible. CherryPy has had support for this since, well, earlier this morning (changeset 549, in the current svn trunk).

As John Udell noted a while back, Adobe Reader uses Range headers to accomplish this if the server supports it. Here's an example .pdf request, and the server's response (many headers omitted for clarity):

GET /mail.pdf HTTP/1.1

200 OK
Accept-Ranges 'bytes'
Content-Length '6786140'
Content-Type 'application/pdf'
Last-Modified 'Mon, 01 Dec 2003 18:13:02 GMT'

On the first request, the server returns a normal 200 response, and begins outputting the file. However, it also outputs the "Accept-Ranges" response header. This tells the client that partial GET requests (using the Range header) will be honored. Therefore, the client tries it, jumping to the PDF's content catalog at the end of the file:

GET /mail.pdf HTTP/1.1
RANGE 'bytes=6633280-6634323,6633278-6633279,6634324-6636107,6669998-6672067,5710727-5712197,

206 Partial Content
Accept-Ranges 'bytes'
Content-Length '158347'
Content-Type 'multipart/byteranges; boundary='

Now our server has returned a different status-code, "206 Partial Content". Since the client requested multiple byteranges, the response body is a multipart/byteranges entity. Each part inside that multipart body has its own Content-Type and Content-Range headers.

Since that worked, Adobe Reader proceeds to read more ranges:

GET /mail.pdf HTTP/1.1
RANGE 'bytes=6626812-6627960,6785258-6786139'

206 Partial Content
Accept-Ranges 'bytes'
Content-Length '2305'
Content-Type 'multipart/byteranges; boundary='

GET /mail.pdf HTTP/1.1
RANGE 'bytes=6636108-6636167,194633-198372,198373-202575'

206 Partial Content
Accept-Ranges 'bytes'
Content-Length '8392'
Content-Type 'multipart/byteranges; boundary='
Last-Modified 'Mon, 01 Dec 2003 18:13:02 GMT'

...and then appears to have finished, after taking some time to process those responses. However, when I scrolled to the last page in the PDF document, Adobe Reader made an additional request:

GET /mail.pdf HTTP/1.1
RANGE 'bytes=6668978-6669997,6672068-6672327,5698663-5699716,5723626-5724706,

206 Partial Content
Accept-Ranges 'bytes'
Content-Length '36370'
Content-Type 'multipart/byteranges; boundary='
Last-Modified 'Mon, 01 Dec 2003 18:13:02 GMT'

So it seems there's quite a bit of partial-retrieval going on, ultimately making the client more responsive from the user's point-of-view.

How to take advantage of partial GET support in CherryPy

If you want to serve static files so that clients can requests portions of them (including resumable downloads!), you only need to use the StaticFilter, which handles Range requests transparently. Here's the script I used to serve mail.pdf:

import cherrypy

class Root: pass
cherrypy.root = Root()

        'server.environment': 'production',
        'staticFilter.on': True,
        'staticFilter.dir': 'static',

Yes, that's really all you need to make an HTTP static file server! The staticFilter.dir (where I saved mail.pdf) is relative to wherever you save the above script. If you'd like it relative to some other absolute path, set that in staticFilter.root.

If you'd like to respond to Range request headers, but you're not serving static files, you can still benefit from CherryPy's core. In cherrypy/_cphttptools, there is a get_ranges(content_length) function which you can use; it examines the current request's Range header, and returns a list of (start, stop) tuples (or just returns None if there's no header). For example, given the Range header:

Range: bytes=30000-40000

The call get_ranges(50000) will return [(30000, 40001)]. Note that we've incremented stop by 1, so that you can use it in a string-slicing operation (byte-ranges are inclusive, but Python's slices have exclusive upper-bounds).

Note also that you need to supply a content-length to get_ranges. It's perfectly valid for a client to request Range: bytes=-500, and expect to receive the last 500 bytes of the resource. So you need to specify the total length in order to do the subtraction.

CherryPy doesn't yet handle the If-Range request header, so feel free to write that and contribute it. ;) ETag support would be nice, too.


Permalink 11:08:42 am, by fumanchu Email , 120 words   English (US)
Categories: CherryPy

Is your code a novel?

Remco, a long-time friend of and contributor to CherryPy, started porting his first app to CP 2.1 today, and had this to say:

<remco>    btw, the code has been cleaned beautifully!!!
[fumanchu] well, thanks
<remco>    respect to all of you who contributed to it
[fumanchu] I tried to make the core process easy to read
<remco>    well, it's still a webserver core, so one has to keep focussed,
<remco>    but compared to 2.0 or prior to that : it reads like a novel! :D
[fumanchu] heh
[fumanchu] 2.0 was a collection of short stories
<remco>    and you can jot that on ur resume

Thanks, I just might do that. :)

<< 1 ... 5 6 7 8 9 10 11 12 13 14 15 ... 17 >>

April 2014
Sun Mon Tue Wed Thu Fri Sat
 << <   > >>
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30      


The requested Blog doesn't exist any more!

XML Feeds

free blog software