« CherryPy for Python 3000Writing High-Efficiency Large Python Systems--Lesson #2: Use nothing but local syslog »

Writing High-Efficiency Large Python Systems--Lesson #3: Banish lazy imports

07/11/08

Permalink 04:16:50 pm, by fumanchu Email , 228 words   English (US)
Categories: WHELPS

Writing High-Efficiency Large Python Systems--Lesson #3: Banish lazy imports

Lazy imports can be done either explicitly, by moving import statements inside functions (instead of at the global level), or by using tools such as LazyImport from egenix. Here's why they suck:

> fetchall (PgSQL:3227)
--> __fetchOneRow (PgSQL:2804)
----> typecast (PgSQL:874)
... 26703 function calls later ...
----< typecast (PgSQL:944): 
      <mx.DateTime.DateTime object for
       '2005-08-15 00:00:00.00' at 2713120>
    3477.321ms

Yes, folks, that single call took 3.4 seconds to run! That would be shorter if I weren't tracing calls, but...ick. Don't make your first customer wait like this in a high-performance app. The solution if you're stuck with lazy imports in code you don't control is to force them to be imported early:

mx.DateTime.Parser.DateFromString('2001-01-01')

Now that same call:

> fetchall (PgSQL:3227)
--> __fetchOneRow (PgSQL:2804)
----> typecast (PgSQL:874)
... 7 function calls later ...
----< typecast (PgSQL:944): 
      <mx.DateTime.DateTime object for
       '2005-08-15 00:00:00.00' at 27cf360>
    1.270ms

That's 1/3815th the number of function calls and 1/2738th the run time. I am not missing decimal points.

Not only is this time-consuming for the first requestor, but lends itself to nasty interactions when a second request starts before the first is done with all the imports. Module import is one of the least-thread-safe parts of almost any app, because people are used to expecting all imports in the main thread at process start.

I'm trying very hard not to rail at length about WSGI frameworks that expect to start up applications during the first HTTP request...but it's so tempting.

5 comments

Comment from: Hoohah [Visitor]

WSGI...would that include the ubiquitous Django?

07/11/08 @ 21:01
Comment from: Graham Dumpleton [Visitor] · http://blog.dscpl.com.au

No, I suspect he doesn't mean Django, or any application framework or toolkit for that matter. I suspect he is talking about hosting systems for WSGI instead and has one in particular in mind. All I will say is WSGIImportScript. The directive exists specifically to allow preloading of code and thus avoid the delay which would otherwise occur the first time a request hits the application. :-)

07/11/08 @ 21:11
Comment from: Evan [Visitor] · http://www.evanfosmark.com/

Wow, I had no idea that they were so inefficient. I'm glad that I haven't been using lazy imports anyways.

07/12/08 @ 15:55
Comment from: PJ Eby [Visitor] · http://thinkingthingsdone.com/

Module-level code is protected via the global import lock (distinct from the GIL), so race conditions for importing a module can't really occur, AFAIK. (If they can, it's a Python bug IMO.)

Regarding lazy import speed, it depends heavily on the implementation involved. I've never seen any big speed differences for my Importing library's lazy imports, but I believe I'm using a totally different strategy than the mx stuff.

07/12/08 @ 16:14
Comment from: Rene Dudfield [Visitor] · http://rene.f0o.com/

hi,

Lazy imports will sometimes be slow because of disk. Doesn't matter about the implementation.

You need to read files from disk, which can take a while.

Often you don't see this in development. However the first people to visit your site will notice it.

This is why 'pre-loaders' and 'pre-cache' are useful too.

Also this could lead to the thundering herds problem. If you restart you might get a whole bunch of people waiting 3 seconds on first request.

07/13/08 @ 18:56

Leave a comment


Your email address will not be revealed on this site.

Your URL will be displayed.

Please enter the phrase "I am a real human." in the textbox above.
(Line breaks become <br />)
(Name, email & website)
(Allow users to contact you through a message form (your email will not be revealed.)
September 2017
Sun Mon Tue Wed Thu Fri Sat
 << <   > >>
          1 2
3 4 5 6 7 8 9
10 11 12 13 14 15 16
17 18 19 20 21 22 23
24 25 26 27 28 29 30

Search

The requested Blog doesn't exist any more!

XML Feeds

free blog tool