Category: CherryPy

Pages: 1 2 3 4 5 6 >>

02/24/11

Permalink 02:58:42 pm, by fumanchu Email , 1214 words   English (US)
Categories: Python, CherryPy

Wow. Does isinstance blow up with ABC's?

Python 2.6.1. Here's a call to "isinstance(value, basestring)":

--[  (_cprequest:782)
--]  (_cprequest:782)  0.044ms

versus "isinstance(value, io.IOBase)":

--[  (_cprequest:791)
----> __instancecheck__ (abc:117)
----. __instancecheck__ (abc:120)
------[  (abc:120)
------]  (abc:120)  0.046ms
----. __instancecheck__ (abc:121)
----. __instancecheck__ (abc:123)
----. __instancecheck__ (abc:124)
----. __instancecheck__ (abc:125)
----. __instancecheck__ (abc:126)
----. __instancecheck__ (abc:127)
----. __instancecheck__ (abc:130)
------> __subclasscheck__ (abc:134)
------. __subclasscheck__ (abc:137)
------. __subclasscheck__ (abc:140)
------. __subclasscheck__ (abc:144)
------. __subclasscheck__ (abc:147)
--------[ ABCMeta.__subclasshook__ (abc:147)
--------] ABCMeta.__subclasshook__ (abc:147)  0.043ms
------. __subclasscheck__ (abc:148)
------. __subclasscheck__ (abc:156)
--------[  (abc:156)
--------]  (abc:156)  0.043ms
------. __subclasscheck__ (abc:160)
------. __subclasscheck__ (abc:165)
--------[ ABCMeta.__subclasses__ (abc:165)
--------] ABCMeta.__subclasses__ (abc:165)  0.045ms
------. __subclasscheck__ (abc:166)
--------[  (abc:166)
----------> __subclasscheck__ (abc:134)
----------. __subclasscheck__ (abc:137)
----------. __subclasscheck__ (abc:140)
----------. __subclasscheck__ (abc:144)
----------. __subclasscheck__ (abc:147)
------------[ ABCMeta.__subclasshook__ (abc:147)
------------] ABCMeta.__subclasshook__ (abc:147)  0.043ms
----------. __subclasscheck__ (abc:148)
----------. __subclasscheck__ (abc:156)
------------[  (abc:156)
------------]  (abc:156)  0.046ms
----------. __subclasscheck__ (abc:160)
----------. __subclasscheck__ (abc:165)
------------[ ABCMeta.__subclasses__ (abc:165)
------------] ABCMeta.__subclasses__ (abc:165)  0.043ms
----------. __subclasscheck__ (abc:166)
------------[  (abc:166)
--------------> __subclasscheck__ (abc:134)
--------------. __subclasscheck__ (abc:137)
--------------. __subclasscheck__ (abc:140)
--------------. __subclasscheck__ (abc:144)
--------------. __subclasscheck__ (abc:147)
----------------[ ABCMeta.__subclasshook__ (abc:147)
----------------] ABCMeta.__subclasshook__ (abc:147)  0.043ms
--------------. __subclasscheck__ (abc:148)
--------------. __subclasscheck__ (abc:156)
----------------[  (abc:156)
----------------]  (abc:156)  0.043ms
--------------. __subclasscheck__ (abc:160)
--------------. __subclasscheck__ (abc:165)
----------------[ ABCMeta.__subclasses__ (abc:165)
----------------] ABCMeta.__subclasses__ (abc:165)  0.042ms
--------------. __subclasscheck__ (abc:170)
----------------[ set.add (abc:170)
----------------] set.add (abc:170)  0.043ms
--------------. __subclasscheck__ (abc:171)
--------------< __subclasscheck__ (abc:171): False 1.690ms
------------]  (abc:166)  1.887ms
----------. __subclasscheck__ (abc:165)
----------. __subclasscheck__ (abc:170)
------------[ set.add (abc:170)
------------] set.add (abc:170)  0.042ms
----------. __subclasscheck__ (abc:171)
----------< __subclasscheck__ (abc:171): False 3.745ms
--------]  (abc:166)  3.952ms
------. __subclasscheck__ (abc:165)
------. __subclasscheck__ (abc:166)
--------[  (abc:166)
----------> __subclasscheck__ (abc:134)
----------. __subclasscheck__ (abc:137)
----------. __subclasscheck__ (abc:140)
----------. __subclasscheck__ (abc:144)
----------. __subclasscheck__ (abc:147)
------------[ ABCMeta.__subclasshook__ (abc:147)
------------] ABCMeta.__subclasshook__ (abc:147)  0.044ms
----------. __subclasscheck__ (abc:148)
----------. __subclasscheck__ (abc:156)
------------[  (abc:156)
------------]  (abc:156)  0.044ms
----------. __subclasscheck__ (abc:160)
----------. __subclasscheck__ (abc:165)
------------[ ABCMeta.__subclasses__ (abc:165)
------------] ABCMeta.__subclasses__ (abc:165)  0.045ms
----------. __subclasscheck__ (abc:166)
------------[  (abc:166)
--------------> __subclasscheck__ (abc:134)
--------------. __subclasscheck__ (abc:137)
--------------. __subclasscheck__ (abc:140)
--------------. __subclasscheck__ (abc:144)
--------------. __subclasscheck__ (abc:147)
----------------[ ABCMeta.__subclasshook__ (abc:147)
----------------] ABCMeta.__subclasshook__ (abc:147)  0.042ms
--------------. __subclasscheck__ (abc:148)
--------------. __subclasscheck__ (abc:156)
----------------[  (abc:156)
----------------]  (abc:156)  0.043ms
--------------. __subclasscheck__ (abc:160)
--------------. __subclasscheck__ (abc:165)
----------------[ ABCMeta.__subclasses__ (abc:165)
----------------] ABCMeta.__subclasses__ (abc:165)  0.043ms
--------------. __subclasscheck__ (abc:166)
----------------[  (abc:166)
------------------> __subclasscheck__ (abc:134)
------------------. __subclasscheck__ (abc:137)
------------------. __subclasscheck__ (abc:140)
------------------. __subclasscheck__ (abc:144)
------------------. __subclasscheck__ (abc:147)
--------------------[ ABCMeta.__subclasshook__ (abc:147)
--------------------] ABCMeta.__subclasshook__ (abc:147)  0.044ms
------------------. __subclasscheck__ (abc:148)
------------------. __subclasscheck__ (abc:156)
--------------------[  (abc:156)
--------------------]  (abc:156)  0.049ms
------------------. __subclasscheck__ (abc:160)
------------------. __subclasscheck__ (abc:165)
--------------------[ ABCMeta.__subclasses__ (abc:165)
--------------------] ABCMeta.__subclasses__ (abc:165)  0.044ms
------------------. __subclasscheck__ (abc:166)
--------------------[  (abc:166)
----------------------> __subclasscheck__ (abc:134)
----------------------. __subclasscheck__ (abc:137)
----------------------. __subclasscheck__ (abc:140)
----------------------. __subclasscheck__ (abc:144)
----------------------. __subclasscheck__ (abc:147)
------------------------[ ABCMeta.__subclasshook__ (abc:147)
------------------------] ABCMeta.__subclasshook__ (abc:147)  0.043ms
----------------------. __subclasscheck__ (abc:148)
----------------------. __subclasscheck__ (abc:156)
------------------------[  (abc:156)
------------------------]  (abc:156)  0.042ms
----------------------. __subclasscheck__ (abc:160)
----------------------. __subclasscheck__ (abc:165)
------------------------[ ABCMeta.__subclasses__ (abc:165)
------------------------] ABCMeta.__subclasses__ (abc:165)  0.042ms
----------------------. __subclasscheck__ (abc:170)
------------------------[ set.add (abc:170)
------------------------] set.add (abc:170)  0.042ms
----------------------. __subclasscheck__ (abc:171)
----------------------< __subclasscheck__ (abc:171): False 1.574ms
--------------------]  (abc:166)  1.772ms
------------------. __subclasscheck__ (abc:165)
------------------. __subclasscheck__ (abc:170)
--------------------[ set.add (abc:170)
--------------------] set.add (abc:170)  0.042ms
------------------. __subclasscheck__ (abc:171)
------------------< __subclasscheck__ (abc:171): False 4.394ms
----------------]  (abc:166)  4.592ms
--------------. __subclasscheck__ (abc:165)
--------------. __subclasscheck__ (abc:166)
----------------[  (abc:166)
------------------> __subclasscheck__ (abc:134)
------------------. __subclasscheck__ (abc:137)
------------------. __subclasscheck__ (abc:140)
------------------. __subclasscheck__ (abc:144)
------------------. __subclasscheck__ (abc:147)
--------------------[ ABCMeta.__subclasshook__ (abc:147)
--------------------] ABCMeta.__subclasshook__ (abc:147)  0.042ms
------------------. __subclasscheck__ (abc:148)
------------------. __subclasscheck__ (abc:156)
--------------------[  (abc:156)
--------------------]  (abc:156)  0.044ms
------------------. __subclasscheck__ (abc:160)
------------------. __subclasscheck__ (abc:165)
--------------------[ ABCMeta.__subclasses__ (abc:165)
--------------------] ABCMeta.__subclasses__ (abc:165)  0.044ms
------------------. __subclasscheck__ (abc:166)
--------------------[  (abc:166)
----------------------> __subclasscheck__ (abc:134)
----------------------. __subclasscheck__ (abc:137)
----------------------. __subclasscheck__ (abc:140)
----------------------. __subclasscheck__ (abc:144)
----------------------. __subclasscheck__ (abc:145)
----------------------< __subclasscheck__ (abc:145): False 0.350ms
--------------------]  (abc:166)  0.553ms
------------------. __subclasscheck__ (abc:165)
------------------. __subclasscheck__ (abc:170)
--------------------[ set.add (abc:170)
--------------------] set.add (abc:170)  0.043ms
------------------. __subclasscheck__ (abc:171)
------------------< __subclasscheck__ (abc:171): False 2.682ms
----------------]  (abc:166)  2.876ms
--------------. __subclasscheck__ (abc:165)
--------------. __subclasscheck__ (abc:170)
----------------[ set.add (abc:170)
----------------] set.add (abc:170)  0.042ms
--------------. __subclasscheck__ (abc:171)
--------------< __subclasscheck__ (abc:171): False 9.633ms
------------]  (abc:166)  9.855ms
----------. __subclasscheck__ (abc:165)
----------. __subclasscheck__ (abc:166)
------------[  (abc:166)
--------------> __subclasscheck__ (abc:134)
--------------. __subclasscheck__ (abc:137)
--------------. __subclasscheck__ (abc:140)
--------------. __subclasscheck__ (abc:144)
--------------. __subclasscheck__ (abc:147)
----------------[ ABCMeta.__subclasshook__ (abc:147)
----------------] ABCMeta.__subclasshook__ (abc:147)  0.042ms
--------------. __subclasscheck__ (abc:148)
--------------. __subclasscheck__ (abc:156)
----------------[  (abc:156)
----------------]  (abc:156)  0.043ms
--------------. __subclasscheck__ (abc:160)
--------------. __subclasscheck__ (abc:165)
----------------[ ABCMeta.__subclasses__ (abc:165)
----------------] ABCMeta.__subclasses__ (abc:165)  0.043ms
--------------. __subclasscheck__ (abc:170)
----------------[ set.add (abc:170)
----------------] set.add (abc:170)  0.042ms
--------------. __subclasscheck__ (abc:171)
--------------< __subclasscheck__ (abc:171): False 1.562ms
------------]  (abc:166)  1.755ms
----------. __subclasscheck__ (abc:165)
----------. __subclasscheck__ (abc:166)
------------[  (abc:166)
--------------> __subclasscheck__ (abc:134)
--------------. __subclasscheck__ (abc:137)
--------------. __subclasscheck__ (abc:140)
--------------. __subclasscheck__ (abc:144)
--------------. __subclasscheck__ (abc:147)
----------------[ ABCMeta.__subclasshook__ (abc:147)
----------------] ABCMeta.__subclasshook__ (abc:147)  0.043ms
--------------. __subclasscheck__ (abc:148)
--------------. __subclasscheck__ (abc:156)
----------------[  (abc:156)
----------------]  (abc:156)  0.043ms
--------------. __subclasscheck__ (abc:160)
--------------. __subclasscheck__ (abc:165)
----------------[ ABCMeta.__subclasses__ (abc:165)
----------------] ABCMeta.__subclasses__ (abc:165)  0.042ms
--------------. __subclasscheck__ (abc:170)
----------------[ set.add (abc:170)
----------------] set.add (abc:170)  0.043ms
--------------. __subclasscheck__ (abc:171)
--------------< __subclasscheck__ (abc:171): False 1.569ms
------------]  (abc:166)  1.772ms
----------. __subclasscheck__ (abc:165)
----------. __subclasscheck__ (abc:166)
------------[  (abc:166)
--------------> __subclasscheck__ (abc:134)
--------------. __subclasscheck__ (abc:137)
--------------. __subclasscheck__ (abc:140)
--------------. __subclasscheck__ (abc:144)
--------------. __subclasscheck__ (abc:147)
----------------[ ABCMeta.__subclasshook__ (abc:147)
----------------] ABCMeta.__subclasshook__ (abc:147)  0.042ms
--------------. __subclasscheck__ (abc:148)
--------------. __subclasscheck__ (abc:156)
----------------[  (abc:156)
----------------]  (abc:156)  0.043ms
--------------. __subclasscheck__ (abc:160)
--------------. __subclasscheck__ (abc:165)
----------------[ ABCMeta.__subclasses__ (abc:165)
----------------] ABCMeta.__subclasses__ (abc:165)  0.042ms
--------------. __subclasscheck__ (abc:170)
----------------[ set.add (abc:170)
----------------] set.add (abc:170)  0.042ms
--------------. __subclasscheck__ (abc:171)
--------------< __subclasscheck__ (abc:171): False 1.647ms
------------]  (abc:166)  1.842ms
----------. __subclasscheck__ (abc:165)
----------. __subclasscheck__ (abc:170)
------------[ set.add (abc:170)
------------] set.add (abc:170)  0.043ms
----------. __subclasscheck__ (abc:171)
----------< __subclasscheck__ (abc:171): False 18.252ms
--------]  (abc:166)  18.443ms
------. __subclasscheck__ (abc:165)
------. __subclasscheck__ (abc:166)
--------[  (abc:166)
----------> __subclasscheck__ (abc:134)
----------. __subclasscheck__ (abc:137)
----------. __subclasscheck__ (abc:140)
----------. __subclasscheck__ (abc:144)
----------. __subclasscheck__ (abc:147)
------------[ ABCMeta.__subclasshook__ (abc:147)
------------] ABCMeta.__subclasshook__ (abc:147)  0.043ms
----------. __subclasscheck__ (abc:148)
----------. __subclasscheck__ (abc:156)
------------[  (abc:156)
------------]  (abc:156)  0.044ms
----------. __subclasscheck__ (abc:160)
----------. __subclasscheck__ (abc:165)
------------[ ABCMeta.__subclasses__ (abc:165)
------------] ABCMeta.__subclasses__ (abc:165)  0.044ms
----------. __subclasscheck__ (abc:166)
------------[  (abc:166)
--------------> __subclasscheck__ (abc:134)
--------------. __subclasscheck__ (abc:137)
--------------. __subclasscheck__ (abc:140)
--------------. __subclasscheck__ (abc:144)
--------------. __subclasscheck__ (abc:147)
----------------[ ABCMeta.__subclasshook__ (abc:147)
----------------] ABCMeta.__subclasshook__ (abc:147)  0.044ms
--------------. __subclasscheck__ (abc:148)
--------------. __subclasscheck__ (abc:156)
----------------[  (abc:156)
----------------]  (abc:156)  0.043ms
--------------. __subclasscheck__ (abc:160)
--------------. __subclasscheck__ (abc:165)
----------------[ ABCMeta.__subclasses__ (abc:165)
----------------] ABCMeta.__subclasses__ (abc:165)  0.044ms
--------------. __subclasscheck__ (abc:166)
----------------[  (abc:166)
------------------> __subclasscheck__ (abc:134)
------------------. __subclasscheck__ (abc:137)
------------------. __subclasscheck__ (abc:140)
------------------. __subclasscheck__ (abc:144)
------------------. __subclasscheck__ (abc:147)
--------------------[ ABCMeta.__subclasshook__ (abc:147)
--------------------] ABCMeta.__subclasshook__ (abc:147)  0.045ms
------------------. __subclasscheck__ (abc:148)
------------------. __subclasscheck__ (abc:156)
--------------------[  (abc:156)
--------------------]  (abc:156)  0.043ms
------------------. __subclasscheck__ (abc:160)
------------------. __subclasscheck__ (abc:165)
--------------------[ ABCMeta.__subclasses__ (abc:165)
--------------------] ABCMeta.__subclasses__ (abc:165)  0.043ms
------------------. __subclasscheck__ (abc:170)
--------------------[ set.add (abc:170)
--------------------] set.add (abc:170)  0.043ms
------------------. __subclasscheck__ (abc:171)
------------------< __subclasscheck__ (abc:171): False 1.624ms
----------------]  (abc:166)  1.867ms
--------------. __subclasscheck__ (abc:165)
--------------. __subclasscheck__ (abc:170)
----------------[ set.add (abc:170)
----------------] set.add (abc:170)  0.041ms
--------------. __subclasscheck__ (abc:171)
--------------< __subclasscheck__ (abc:171): False 3.866ms
------------]  (abc:166)  4.063ms
----------. __subclasscheck__ (abc:165)
----------. __subclasscheck__ (abc:170)
------------[ set.add (abc:170)
------------] set.add (abc:170)  0.043ms
----------. __subclasscheck__ (abc:171)
----------< __subclasscheck__ (abc:171): False 5.968ms
--------]  (abc:166)  6.159ms
------. __subclasscheck__ (abc:165)
------. __subclasscheck__ (abc:170)
--------[ set.add (abc:170)
--------] set.add (abc:170)  0.042ms
------. __subclasscheck__ (abc:171)
------< __subclasscheck__ (abc:171): False 31.110ms
----< __instancecheck__ (abc:130): False 32.160ms
--]  (_cprequest:791)  32.350ms

11/19/10

Permalink 01:08:45 am, by fumanchu Email , 1007 words   English (US)
Categories: Python, CherryPy

logging.statistics

Statistics about program operation are an invaluable monitoring and debugging tool. How many requests are being handled per second, how much of various resources are in use, how long we've been up. Unfortunately, the gathering and reporting of these critical values is usually ad-hoc. It would be nice if we had 1) a centralized place for gathering statistical performance data, 2) a system for extrapolating that data into more useful information, and 3) a method of serving that information to both human investigators and monitoring software. I've got a proposal. Let's examine each of those points in more detail.

Data Gathering

Just as Python's logging module provides a common importable for gathering and sending messages, statistics need a similar mechanism, and one that does not require each package which wishes to collect stats to import a third-party module. Therefore, we choose to re-use the logging module by adding a statistics object to it.

That logging.statistics object is a nested dict:

import logging
if not hasattr(logging, 'statistics'): logging.statistics = {}

It is not a custom class, because that would 1) require apps to import a third-party module in order to participate, 2) inhibit innovation in extrapolation approaches and in reporting tools, and 3) be slow. There are, however, some specifications regarding the structure of the dict.

    {
   +----"SQLAlchemy": {
   |        "Inserts": 4389745,
   |        "Inserts per Second":
   |            lambda s: s["Inserts"] / (time() - s["Start"]),
   |  C +---"Table Statistics": {
   |  o |        "widgets": {-----------+
 N |  l |            "Rows": 1.3M,      | Record
 a |  l |            "Inserts": 400,    |
 m |  e |        },---------------------+
 e |  c |        "froobles": {
 s |  t |            "Rows": 7845,
 p |  i |            "Inserts": 0,
 a |  o |        },
 c |  n +---},
 e |        "Slow Queries":
   |            [{"Query": "SELECT * FROM widgets;",
   |              "Processing Time": 47.840923343,
   |              },
   |             ],
   +----},
    }

The logging.statistics dict has strictly 4 levels. The topmost level is nothing more than a set of names to introduce modularity. If SQLAlchemy wanted to participate, it might populate the item logging.statistics['SQLAlchemy'], whose value would be a second-layer dict we call a "namespace". Namespaces help multiple emitters to avoid collisions over key names, and make reports easier to read, to boot. The maintainers of SQLAlchemy should feel free to use more than one namespace if needed (such as 'SQLAlchemy ORM').

Each namespace, then, is a dict of named statistical values, such as 'Requests/sec' or 'Uptime'. You should choose names which will look good on a report: spaces and capitalization are just fine.

In addition to scalars, values in a namespace MAY be a (third-layer) dict, or a list, called a "collection". For example, the CherryPy StatsTool keeps track of what each worker thread is doing (or has most recently done) in a 'Worker Threads' collection, where each key is a thread ID; each value in the subdict MUST be a fourth dict (whew!) of statistical data about
each thread. We call each subdict in the collection a "record". Similarly, the StatsTool also keeps a list of slow queries, where each record contains data about each slow query, in order.

Values in a namespace or record may also be functions, which brings us to:

Extrapolation

def extrapolate_statistics(scope):
    """Return an extrapolated copy of the given scope."""
    c = {}
    for k, v in scope.items():
        if isinstance(v, dict):
            v = extrapolate_statistics(v)
        elif isinstance(v, (list, tuple)):
            v = [extrapolate_statistics(record) for record in v]
        elif callable(v):
            v = v(scope)
        c[k] = v
    return c

The collection of statistical data needs to be fast, as close to unnoticeable as possible to the host program. That requires us to minimize I/O, for example, but in Python it also means we need to minimize function calls. So when you are designing your namespace and record values, try to insert the most basic scalar values you already have on hand.

When it comes time to report on the gathered data, however, we usually have much more freedom in what we can calculate. Therefore, whenever reporting tools fetch the contents of logging.statistics for reporting, they first call extrapolate_statistics (passing the whole statistics dict as the only argument). This makes a deep copy of the statistics dict so that the reporting tool can both iterate over it and even change it without harming the original. But it also expands any functions in the dict by calling them. For example, you might have a 'Current Time' entry in the namespace with the value "lambda scope: time.time()". The "scope" parameter is the current namespace dict (or record, if we're currently expanding one of those instead), allowing you access to existing static entries. If you're truly evil, you can even modify more than one entry at a time.

However, don't try to calculate an entry and then use its value in further extrapolations; the order in which the functions are called is not guaranteed. This can lead to a certain amount of duplicated work (or a redesign of your schema), but that's better than complicating the spec.

After the whole thing has been extrapolated, it's time for:

Reporting

A reporting tool would grab the logging.statistics dict, extrapolate it all, and then transform it to (for example) HTML for easy viewing, or JSON for processing by Nagios etc (and because JSON will be a popular output format, you should seriously consider using Python's time module for datetimes and arithmetic, not the datetime module). Each namespace might get its own header and attribute table, plus an extra table for each collection. This is NOT part of the statistics specification; other tools can format how they like.

Turning Collection Off

It is recommended each namespace have an "Enabled" item which, if False, stops collection (but not reporting) of statistical data. Applications SHOULD provide controls to pause and resume collection by setting these entries to False or True, if present.

Usage

    import logging
    # Initialize the repository
    if not hasattr(logging, 'statistics'): logging.statistics = {}
    # Initialize my namespace
    mystats = logging.statistics.setdefault('My Stuff', {})
    # Initialize my namespace's scalars and collections
    mystats.update({
        'Enabled': True,
        'Start Time': time.time(),
        'Important Events': 0,
        'Events/Second': lambda s: (
            (s['Important Events'] / (time.time() - s['Start Time']))),
        })
    ...
    for event in events:
        ...
        # Collect stats
        if mystats.get('Enabled', False):
            mystats['Important Events'] += 1

03/11/10

Permalink 08:29:40 pm, by fumanchu Email , 11 words   English (US)
Categories: IT, Python, CherryPy

Zen of CherryPy video

My PyCon 2010 talk video is up. Enjoy: The Zen of CherryPy

07/14/09

Permalink 08:10:47 pm, by fumanchu Email , 43 words   English (US)
Categories: IT, Python, Dejavu, CherryPy

The Ronacher Manifesto

Link: http://lucumr.pocoo.org/2009/7/14/free-vs-free

I heartily agree with the bold bits at least:

So dear users: Use my stuff, have fun with it. And letting me know that you're doing is the best reward I can think of. And if you can contribute patches, that's even better.

05/05/09

Permalink 10:00:00 am, by fumanchu Email , 22 words   English (US)
Categories: IT, CherryPy

Best of PyCon 2009

Toshio Kuratomi's How to Build Applications Linux Distributions will Package. As a web framework dev, this was priceless.

04/07/09

Permalink 10:06:00 am, by fumanchu Email , 132 words   English (US)
Categories: IT, Python, Dejavu, CherryPy

PyCon Presentations

For PyCon 2009, I'm giving two talks! One on extending CherryPy and one on the innards of Dejavu/GeniuSQL. I think I've finally reduced my talks to the required time slots (I could easily have made 4-hour talks for each ;) and posted my presentations:

Use the arrow keys or mouse-click to proceed through them. The images don't load as fast over the network as they will when I present, so be patient if you preview them yourself. Also, try to use 1024 x 768 fullscreen--they're laid out specifically for that resolution.

Update: video is now available thanks to the great people who put on PyCon:

09/08/08

Permalink 11:36:06 am, by fumanchu Email , 132 words   English (US)
Categories: IT, CherryPy, WSGI

Resources are concepts for the client

...not objects on the server. Roy Fielding explains yet again:

Web architects must understand that resources are just consistent mappings from an identifier to some set of views on server-side state. If one view doesn’t suit your needs, then feel free to create a different resource that provides a better view (for any definition of “better”). These views need not have anything to do with how the information is stored on the server, or even what kind of state it ultimately reflects. It just needs to be understandable (and actionable) by the recipient.

I have found this to be the single most-misunderstood aspect of HTTP. Too many people conceive of URI's as just names for files or database objects. They can be so much more.

07/25/08

Permalink 01:20:56 am, by fumanchu Email , 243 words   English (US)
Categories: Python, CherryPy

CherryPy for Python 3000

I'm categorically rejecting the 2to3 approach--for myself anyway. If you think it would help, feel free to:

  1. "upgrade" CP to 2.6, which AFAICT means ensuring it will no longer work in 2.5 or previous versions
  2. turn on the 3k warning
  3. import-and-fix until you don't get any warnings
  4. run-tests-and-fix until you don't get any warnings
  5. run 2to3
  6. import-and-fix until you don't get any errors
  7. run-tests-and-fix until you don't get any errors
  8. wait for bug reports

Me, I'd rather just drop cherrypy/ into 3k and skip steps 1-5.

Changes I had to make so far (http://www.cherrypy.org/changeset/2029):

  • (4) urlparse -> urllib.parse
  • (24) "except (ExcA, ExcB):" -> "except ExcA, ExcB:"
  • (30) "except ExcClass, x:" -> "except ExcClass as x"
  • (22) u"" -> ""
  • (1) BaseHTTPServer -> http.server
  • (1) rfc822 -> email.utils
  • (4) md5.new() -> hashlib.md5()
  • (3) sha.new() -> hashlib.sha1()
  • (3) urllib2 -> urllib
  • (28) StringIO -> io
  • (1) func.func_code -> func.code
  • (6) Cookie -> http.cookies
  • (3) ConfigParser -> configparser
  • (1) rfc822._monthnames -> email._parseaddr._monthnames
  • (105) print -> print()
  • (35) httplib -> http.client
  • (22) basestring -> (str, bytes)
  • (12) items() -> list(items())
  • (46) iteritems() -> items()
  • (11) Thread.get/setName -> get/set_name
  • (1) exec "" -> exec("")
  • (1) 0777 -> 0o777
  • (1) Queue -> queue
  • (1) urllib.unquote -> urllib.parse.unquote

At the moment, I'm a bit blocked importing wsgiserver--we had a nonblocking version of makefile that subclassed the old socket._fileobject class. Looks like the whole socket implementation has changed (and much of it pushed down into C). Not looking forward to reimplementing that.

06/11/08

Permalink 08:56:58 pm, by fumanchu Email , 335 words   English (US)
Categories: Python, CherryPy

Tracking memory leaks with Dowser

Marius Gedminas just wrote a post on memory leaks. He could have used Dowser to find the leak more easily, I'll bet.

Dowser is a CherryPy application for monitoring and managing object references in your Python program. Because CherryPy runs everything (even the listening HTTP socket) in its own threads, it's a snap to include Dowser in any Python process. Dowser is also very lightweight (because CherryPy is). Here's how I added it to a Twisted project we're using at work:

...
from twisted.application import service
application = service.Application("My Server")
s.setServiceParent(application)

import cherrypy
from misc import dowser
cherrypy.config.update({'server.socket_port': 8088})
cherrypy.tree.mount(dowser.Root())
cherrypy.engine.autoreload.unsubscribe()
# Windows only
cherrypy._console_control_handler.unsubscribe()
cherrypy.engine.start()

from twisted.internet import reactor
reactor.addSystemEventTrigger('after', 'shutdown', cherrypy.engine.exit)

The lines before 'import cherrypy' already existed and are here just for context (this is a Twisted service.tac module). Let's quickly discuss the new code:

  1. import cherrypy and dowser. You don't have to stick dowser into a 'misc' folder; that's just how I checked it out from svn.
  2. Set the port you want CherryPy to listen on; pick a port your app isn't already using if it's a TCP server.
  3. Mount the dowser root.
  4. Turn off the CherryPy autoreloader, and the Ctrl-C handler if you're on Windows. I should really turn that off by default in CP. :/
  5. Start the engine, which starts listening on the port in a new thread among other things.
  6. Tell Twisted to stop CherryPy when it stops.

Then browse to http://localhost:8088/ and you'll see pretty sparklines of all the objects. Change the URL to http://localhost:8088/?floor=20 to see graphs for only those objects which have 20 or more objects.

Then, just click on the 'TRACE' links to get lots more information about each object. See the Dowser wiki page for more details and screenshots.

10/02/07

Permalink 12:55:58 am, by fumanchu Email , 159 words   English (US)
Categories: CherryPy, WSGI

CherryPy 3 request_queue_size

Well, that was instructive. Leaving server.request_queue_size at the default 5:

C:\Python24\Lib\site-packages>python cherrypy\test\benchmark.py
Starting CherryPy app server...
Started in 1.10800004005 seconds

Client Thread Report (1000 requests, 14 byte response body, 10 server threads):

threads | Completed | Failed | req/sec | msec/req | KB/sec |
     10 |      1000 |      0 |  736.81 |    1.357 | 119.36 |
     20 |      1000 |      0 |  436.07 |    2.293 |  70.64 |
     30 |      1000 |      0 |  348.38 |    2.870 |  56.44 |
     40 |      1000 |      0 |  233.10 |    4.290 |  37.76 |
     50 |      1000 |      0 |  296.77 |    3.370 |  48.08 |
Average |    1000.0 |    0.0 | 410.226 |    2.836 | 66.456 |

Client Thread Report (1000 requests, 14 bytes via staticdir, 10 server threads):

threads | Completed | Failed | req/sec | msec/req | KB/sec |
     10 |      1000 |      0 |  421.73 |    2.371 |  87.30 |
     20 |      1000 |      0 |  374.87 |    2.668 |  77.60 |
     30 |      1000 |      0 |  306.71 |    3.260 |  63.49 |
     40 |      1000 |      0 |  240.08 |    4.165 |  49.70 |
     50 |      1000 |      0 |  170.03 |    5.881 |  35.20 |
Average |    1000.0 |    0.0 | 302.684 |    3.669 | 62.658 |

Size Report (1000 requests, 50 client threads, 10 server threads):

    bytes | Completed | Failed | req/sec | msec/req |   KB/sec |
       10 |      1000 |      0 |  187.98 |    5.320 |    29.70 |
      100 |      1000 |      0 |  207.45 |    4.820 |    51.45 |
     1000 |      1000 |      0 |  186.89 |    5.351 |   210.81 |
    10000 |      1000 |      0 |  228.12 |    4.384 |  2262.07 |
   100000 |      1000 |      0 |  245.60 |    4.072 | 24022.01 |
100000000 |      1000 |     10 |   20.83 |   48.001 | 20358.12 |

Upping server.request_queue_size to 128:

C:\Python24\Lib\site-packages>python cherrypy\test\benchmark.py
Starting CherryPy app server...
Started in 1.10700011253 seconds

Client Thread Report (1000 requests, 14 byte response body, 10 server threads):

threads | Completed | Failed | req/sec | msec/req |  KB/sec |
     10 |      1000 |      0 |  745.38 |    1.342 |  120.75 |
     20 |      1000 |      0 |  772.32 |    1.295 |  125.12 |
     30 |      1000 |      0 |  654.11 |    1.529 |  105.97 |
     40 |      1000 |      0 |  929.02 |    1.076 |  150.50 |
     50 |      1000 |      0 |  641.03 |    1.560 |  103.85 |
Average |    1000.0 |    0.0 | 748.372 |   1.3604 | 121.238 |

Client Thread Report (1000 requests, 14 bytes via staticdir, 10 server threads):

threads | Completed | Failed | req/sec | msec/req |  KB/sec |
     10 |      1000 |      0 |  547.89 |    1.825 |  113.41 |
     20 |      1000 |      0 |  588.10 |    1.700 |  121.74 |
     30 |      1000 |      0 |  704.42 |    1.420 |  145.82 |
     40 |      1000 |      0 |  547.89 |    1.825 |  113.41 |
     50 |      1000 |      0 |  516.96 |    1.934 |  107.01 |
Average |    1000.0 |    0.0 | 581.052 |   1.7408 | 120.278 |

Size Report (1000 requests, 50 client threads, 10 server threads):

    bytes | Completed | Failed | req/sec | msec/req |   KB/sec |
       10 |      1000 |      0 |  622.35 |    1.607 |    98.33 |
      100 |      1000 |      0 |  604.74 |    1.654 |   149.37 |
     1000 |      1000 |      0 |  667.74 |    1.498 |   752.54 |
    10000 |      1000 |      0 |  890.31 |    1.123 |  8837.25 |
   100000 |      1000 |      0 |  728.44 |    1.373 | 71247.09 |
100000000 |      1000 |    202 |   12.81 |   78.094 |     None |

1 2 3 4 5 6 >>

August 2017
Sun Mon Tue Wed Thu Fri Sat
 << <   > >>
    1 2 3 4 5
6 7 8 9 10 11 12
13 14 15 16 17 18 19
20 21 22 23 24 25 26
27 28 29 30 31    

Search

The requested Blog doesn't exist any more!

XML Feeds

free blog software