Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 128

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 134

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 141

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 169

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 199

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 205

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 233

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 248

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 254

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 267

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php on line 595

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/files/model/_file.funcs.php on line 559

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_connect_db.inc.php on line 29

Strict Standards: Declaration of UserSettings::get() should be compatible with AbstractSettings::get($col_key1, $col_key2 = NULL, $col_key3 = NULL) in /home/fumanchu/webapps/b2/blogs/inc/users/model/_usersettings.class.php on line 208

Strict Standards: Declaration of UserSettings::set() should be compatible with AbstractSettings::set() in /home/fumanchu/webapps/b2/blogs/inc/users/model/_usersettings.class.php on line 208

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_core/_param.funcs.php on line 1692

Warning: Cannot modify header information - headers already sent by (output started at /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php:128) in /home/fumanchu/webapps/b2/blogs/inc/sessions/model/_session.class.php on line 219

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/generic/model/_genericelement.class.php on line 109

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_core/model/dataobjects/_dataobject.class.php on line 428

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_core/model/dataobjects/_dataobject.class.php on line 437

Strict Standards: Declaration of Blog::set() should be compatible with DataObject::set($parname, $parvalue, $make_null = false) in /home/fumanchu/webapps/b2/blogs/inc/collections/model/_blog.class.php on line 1886

Strict Standards: Declaration of Group::set() should be compatible with DataObject::set($parname, $parvalue, $make_null = false) in /home/fumanchu/webapps/b2/blogs/inc/users/model/_group.class.php on line 553

Strict Standards: Declaration of User::dbdelete() should be compatible with DataObject::dbdelete() in /home/fumanchu/webapps/b2/blogs/inc/users/model/_user.class.php on line 1343

Strict Standards: Declaration of User::set() should be compatible with DataObject::set($parname, $parvalue, $make_null = false) in /home/fumanchu/webapps/b2/blogs/inc/users/model/_user.class.php on line 1343

Strict Standards: Declaration of Filetype::set() should be compatible with DataObject::set($parname, $parvalue, $make_null = false) in /home/fumanchu/webapps/b2/blogs/inc/files/model/_filetype.class.php on line 192

Strict Standards: Declaration of Comment::set() should be compatible with DataObject::set($parname, $parvalue, $make_null = false) in /home/fumanchu/webapps/b2/blogs/inc/comments/model/_comment.class.php on line 1334

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/_blog_main.inc.php on line 412

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/items/model/_itemlist.class.php on line 483

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/items/model/_itemlistlight.class.php on line 119

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/items/model/_itemlistlight.class.php on line 838

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/items/model/_item.class.php on line 1411

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/items/model/_item.class.php on line 1414

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/items/model/_item.class.php on line 3000

Strict Standards: Declaration of ItemListLight::query() should be compatible with Results::query($create_default_cols_if_needed = true, $append_limit = true, $append_order_by = true, $query_title = 'Results::Q...') in /home/fumanchu/webapps/b2/blogs/inc/items/model/_itemlistlight.class.php on line 48

Strict Standards: Declaration of GenericCategoryCache::clear() should be compatible with DataObjectCache::clear($keep_shadow = false) in /home/fumanchu/webapps/b2/blogs/inc/generic/model/_genericcategorycache.class.php on line 375

Strict Standards: Declaration of GenericCategory::set() should be compatible with DataObject::set($parname, $parvalue, $make_null = false) in /home/fumanchu/webapps/b2/blogs/inc/generic/model/_genericcategory.class.php on line 133

Strict Standards: Declaration of CollectionSettings::_load() should be compatible with AbstractSettings::_load($arg1 = NULL, $arg2 = NULL, $arg3 = NULL) in /home/fumanchu/webapps/b2/blogs/inc/collections/model/_collsettings.class.php on line 130

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 236

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 236

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 236

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 236

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 236

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 236

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 236

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 236

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 236

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 236

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Notice: Array to string conversion in /home/fumanchu/webapps/b2/blogs/inc/_core/_param.funcs.php on line 1161

Notice: Array to string conversion in /home/fumanchu/webapps/b2/blogs/inc/_core/_param.funcs.php on line 1161

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Warning: Cannot modify header information - headers already sent by (output started at /home/fumanchu/webapps/b2/blogs/inc/_main.inc.php:128) in /home/fumanchu/webapps/b2/blogs/inc/skins/_skin.funcs.php on line 379
Category: WHELPS - The Hand of FuManChu

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/widgets/model/_widget.class.php on line 451

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/inc/widgets/model/_widget.class.php on line 547

Strict Standards: Declaration of ComponentWidget::set() should be compatible with DataObject::set($parname, $parvalue, $make_null = false) in /home/fumanchu/webapps/b2/blogs/inc/widgets/model/_widget.class.php on line 722

Category: WHELPS

07/11/08


Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334
Permalink 04:16:50 pm, by fumanchu Email , 228 words   English (US)
Categories: WHELPS


Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334
Writing High-Efficiency Large Python Systems--Lesson #3: Banish lazy imports


Strict Standards: Declaration of markdown_extra_plugin::GetDefaultSettings() should be compatible with Plugin::GetDefaultSettings(&$params) in /home/fumanchu/webapps/b2/blogs/plugins/_markdown_extra.plugin.php on line 84

Strict Standards: Declaration of smilies_plugin::GetDefaultSettings() should be compatible with Plugin::GetDefaultSettings(&$params) in /home/fumanchu/webapps/b2/blogs/plugins/_smilies.plugin.php on line 398

Strict Standards: Declaration of smilies_plugin::GetDefaultUserSettings() should be compatible with Plugin::GetDefaultUserSettings(&$params) in /home/fumanchu/webapps/b2/blogs/plugins/_smilies.plugin.php on line 398

Strict Standards: Declaration of auto_p_plugin::GetDefaultSettings() should be compatible with Plugin::GetDefaultSettings(&$params) in /home/fumanchu/webapps/b2/blogs/plugins/_auto_p.plugin.php on line 633

Lazy imports can be done either explicitly, by moving import statements inside functions (instead of at the global level), or by using tools such as LazyImport from egenix. Here's why they suck:

> fetchall (PgSQL:3227)
--> __fetchOneRow (PgSQL:2804)
----> typecast (PgSQL:874)
... 26703 function calls later ...
----< typecast (PgSQL:944): 
      <mx.DateTime.DateTime object for
       '2005-08-15 00:00:00.00' at 2713120>
    3477.321ms

Yes, folks, that single call took 3.4 seconds to run! That would be shorter if I weren't tracing calls, but...ick. Don't make your first customer wait like this in a high-performance app. The solution if you're stuck with lazy imports in code you don't control is to force them to be imported early:

mx.DateTime.Parser.DateFromString('2001-01-01')

Now that same call:

> fetchall (PgSQL:3227)
--> __fetchOneRow (PgSQL:2804)
----> typecast (PgSQL:874)
... 7 function calls later ...
----< typecast (PgSQL:944): 
      <mx.DateTime.DateTime object for
       '2005-08-15 00:00:00.00' at 27cf360>
    1.270ms

That's 1/3815th the number of function calls and 1/2738th the run time. I am not missing decimal points.

Not only is this time-consuming for the first requestor, but lends itself to nasty interactions when a second request starts before the first is done with all the imports. Module import is one of the least-thread-safe parts of almost any app, because people are used to expecting all imports in the main thread at process start.

I'm trying very hard not to rail at length about WSGI frameworks that expect to start up applications during the first HTTP request...but it's so tempting.


Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/skins/_item_feedback.inc.php on line 156

07/03/08

Permalink 05:37:31 pm, by fumanchu Email , 319 words   English (US)
Categories: WHELPS

Writing High-Efficiency Large Python Systems--Lesson #2: Use nothing but local syslog

You want to log everything, but you'll find that even in the simplest requests with the fastest response times, a simple file-based access log can add 10% to your response time (which usually means ~91% as many requests per second). The fastest substitute we've found for file-based logging in Python is syslog. Here's how easy it is:

import syslog
syslog.syslog(facility | priority, msg)

Nothing's faster, at least nothing that doesn't require you telling Operations to compile a new C module on their production servers.

"But wait!" you say, "Python's builtin logging module has a SysLogHandler! Use that!" Well, no. There are two reasons why not. First, because Python's logging module in general is bog-slow--too slow for high-efficiency apps. It can make many function calls just to decide it's not going to log a message. Second, the SysLogHandler in the stdlib uses a UDP socket by default. You can pass it a string for the address (probably '/dev/log') and it will use a UNIX socket just like syslog.syslog, but it'll still do it in Python, not C, and you still have all the logging module overhead.

Here's a SysLogLibHandler if you're stuck with the stdlib logging module:

class SysLogLibHandler(logging.Handler):
    """A logging handler that emits messages to syslog.syslog."""
    priority_map = {
        10: syslog.LOG_NOTICE, 
        20: syslog.LOG_NOTICE, 
        30: syslog.LOG_WARNING, 
        40: syslog.LOG_ERR, 
        50: syslog.LOG_CRIT, 
        0: syslog.LOG_NOTICE, 
        }

    def __init__(self, facility):
        self.facility = facility
        logging.Handler.__init__(self)

    def emit(self, record):
        syslog.syslog(self.facility | self.priority_map[record.levelno],
                      self.format(record))

I suggest using syslog.LOCAL0 - syslog.LOCAL7 for the facility arg. If you're writing a server, use one facility for access log messages and a different one for error/debug logs. Then you can configure syslogd to handle them differently (e.g., send them to /var/log/myapp/access.log and /var/log/myapp/error.log).


Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/skins/_item_feedback.inc.php on line 156
Permalink 05:02:59 pm, by fumanchu Email , 189 words   English (US)
Categories: WHELPS

Writing High-Efficiency Large Python Systems--Lesson #1: Transactions in tests

Don't write your test suite to create and destroy databases for each run. Instead, make each test method start a transaction and roll it back. We just made that move at work on a DAL project, and the test suite went from 500+ seconds to run the whole thing down to around 100. It also allowed us to remove a lot of "undo" code in the tests.

This means ensuring your test helpers always connect to their databases on the same connection (transactions are connection-specific). If you're using a connection pool where leased conns are bound to each thread, this means rewriting tests that start new threads (or leaving them "the old way"; that is, create/drop). It also means that, rather than running slightly different .sql files per test or module, you instead have a base of data and allow each test to add other data as needed. If your rollbacks work, these can't pollute other tests.

Obviously, this is much harder if you're doing integration testing of sharded systems and the like. But for application logic, it'll save you a lot of headache to do this from the start.


Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/skins/_item_feedback.inc.php on line 156

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/plugins/_calendar.plugin.php on line 190

Deprecated: Assigning the return value of new by reference is deprecated in /home/fumanchu/webapps/b2/blogs/plugins/_calendar.plugin.php on line 905

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 334
November 2020
Sun Mon Tue Wed Thu Fri Sat
 << <   > >>
1 2 3 4 5 6 7
8 9 10 11 12 13 14
15 16 17 18 19 20 21
22 23 24 25 26 27 28
29 30          

Search

The requested Blog doesn't exist any more!

XML Feeds


Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 223
powered by b2evolution

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 304

Deprecated: Non-static method Hitlist::dbprune() should not be called statically, assuming $this from incompatible context in /home/fumanchu/webapps/b2/blogs/inc/sessions/model/_hit.class.php on line 531

Warning: Creating default object from empty value in /home/fumanchu/webapps/b2/blogs/inc/settings/model/_abstractsettings.class.php on line 304