Pages: << 1 2 3 4 5 6 7 8 9 10 11 >>
Jeff Atwood, having read Steve Yegge's Tour de Babel, write the following:
I've often thought that sucking less every year is how humble programmers improve. You should be unhappy with code you wrote a year ago. If you aren't, that means either A) you haven't learned anything in a year,
your code can't be improved, or C) you never revisit old code. All of these are the kiss of death for software developers.
They are also, therefore, the kiss of death for open-source software projects.
And yet, somehow, both new and experienced developers look at new and even gently-used open-source software efforts and expect them to be perfect. Not even, "this works as advertised" perfect, but "this is exactly how I would have written it" perfect. Never mind that, if they had written it, it would suck in a year, too.
This has been bothering me for a long time, and boils down to this maxim: "Everything's hard until you've done it once." Computation is hard. Search is hard. Concurrency is hard. Interfaces are hard. It's easy for smart people to believe that what they know today has always been known by everyone, and to conclude that, if you don't know it yet, you are a total dunce. Don't fall into that trap, people.
One of the things I like best about the Python community in particular is a healthy dose of humility. Let's keep it going.
While at PyCon in Dallas this weekend, I got a chance to hear David Creemer talk about how he's using CherryPy (among lots of other tools) to deliver a site that's getting 250,000 hits a day before it's even been officially launched. He mentioned during the "lightning" (5 minute) talk that, of his entire toolkit, CherryPy and SQLObject were two of the tools that "mostly worked, except..." I spoke with him after the talk about his concerns, and the big CherryPy issue was dispatching: he prefers a Routes-style dispatch mechanism, which makes changes to the design easier.
He had previously posted his cherrypy+routes script, which I read but hadn't done anything about. I mentioned to him yesterday that it might be better, when overriding the dispatch mechanism, to do so in a custom Request class, rather than in a single exposed default method on the cherrypy tree. Here's a first crack at what that would look like; I haven't tested it but it's more to get the idea of custom Request classes out there than to be a working patch
import urllib
import cherrypy
from cherrypy import _cphttptools
import routes
mapper = routes.Mapper()
controllers = {}
def redirect( url ):
raise cherrypy.HTTPRedirect( url )
def mapConnect( name, route, controller, **kwargs ):
controllers[ name ] = controller
mapper.connect( name, route, controller=name, **kwargs )
def mapFinalize():
mapper.create_regs( controllers.keys() )
def URL( name, query = None, doseq = None, **kwargs ):
uri = routes.url_for( name, **kwargs )
if not uri:
return "/UNKNOWN-%s" % name
if query:
uri += '?' + urllib.urlencode(query, doseq)
return uri
class RoutesRequest(_cphttptools.Request):
def main(self, path=None):
"""Obtain and set cherrypy.response.body from a page handler."""
if path is None:
path = self.object_path
page_handler = self.mapPathToObject(path)
virtual_path = path.split("/")
# Decode any leftover %2F in the virtual_path atoms.
virtual_path = [x.replace("%2F", "/") for x in virtual_path if x]
kwargs = self.params.copy()
kwargs.update( cherrypy.request.mapper_dict )
try:
body = page_handler(*virtual_path, **kwargs)
except Exception, x:
x.args = x.args + (page_handler,)
raise
cherrypy.response.body = body
def mapPathToObject(self, objectpath):
"""For path, return the corresponding exposed callable (or raise NotFound).
path should be a "relative" URL path, like "/app/a/b/c". Leading and
trailing slashes are ignored.
"""
# tell routes to use the cherrypy threadlocal object
config = routes.request_config()
if hasattr(config, 'using_request_local'):
config.request_local = lambda: self
config = routes.request_config()
# hook up the routes variables for this request
config.mapper = mapper
config.host = self.headerMap['Host']
config.protocol = self.scheme
config.redirect = redirect
config.mapper_dict = mapper.match( objectpath )
if config.mapper_dict:
c = config.mapper_dict.pop( 'controller', None )
if c:
controller = controllers[c]
# we have a controller, now emulate cherrypy's index/default/callable semantics:
action = config.mapper_dict.pop( 'action', 'index' )
meth = getattr( controller, action, None )
if not meth:
meth = getattr( controller, 'default', None )
if not meth and callable( controller ) and action == 'index' :
meth = controller
if meth and getattr( meth, 'exposed', False ):
return meth
raise cherrypy.NotFound( objectpath )
# 'authui' is a module with login/logout functions
mapConnect( name = 'home', route = '', controller = home )
mapConnect( name = 'auth', route = 'auth/:action', controller = authui,
requirements = dict( action='(login|logout)' ) )
mapFinalize()
cherrypy.server.request_class = RoutesRequest
cherrypy.server.start()
Look, Ma, no root!
I'm extremely pleased to announce the release of Dejavu 1.4.0, a pure-Python Object-Relational Mapping library. Dejavu allows you to create, query, and manage persistent data using your existing knowledge of Python programming.
Dejavu is in the Public Domain, and you may use it anywhere with no obligation whatsoever.
User documentation and a full Trac site are available at: http://projects.amor.org/dejavu/
The job posting is pretty tame: we need a Python web developer. But I thought I'd add my personal point-of-view, and say that we really mean "developer" and not just "coder". You'd be responsible for producing working web apps, but that involves a lot of design work and architectural decision-making.
You would also be expected to contribute to the CherryPy HTTP framework and to Dejavu (my Python ORM), since I'm a core dev on both those projects and use them heavily already. In other words, if you have or want exposure to the full stack of modern web development challenges, this is the job for you. You'll be a full member of an IT team of 3 serving an energetic staff of 50.
You'll also get something that's hard to find in most programming jobs: warm fuzzies. We build homes for the poor in Mexico, simultaneously "building" the church in Mexico, the U.S., Canada, and elsewhere. We are not on the cutting-edge of world missions--we are defining that edge. If you've been thinking about "doing more for Jesus", but would rather write code than dig ditches in Uganda, give us a call (619-662-1200 ext 11).
After more than a year since 1.3 was released, I'm just about ready to officially release Dejavu 1.4! In addition to bugfixes, there are some major new features:
Sandbox.recall(classes, expr)
whether you're querying a single class or multiple ones.inv = box.Invoice(13)
.As you can see, a year's worth of work. Feel free to kick the tires on all the new stuff. I should bless a release candidate in early January.
I just dumped a first crack at a Schema class on the trunk. Test code is here (search for 'schema'), docs are here. I haven't written anything like this before, so if anyone has recommendations or warnings about the direction it's heading, now is the time to speak up (before 1.4 is officially released )!
Basic design: there's a dejavu.Schema class which your app can subclass. Whenever you need to change the underlying database (or other persistence mechanism) schema of your app, you write a new upgrade_to_X
method, where X is an incrementing version number. Each such method contains the commands which will upgrade an installation from (X - 1) to X.
At runtime, you call MySchema.upgrade()
, and each deployment will run any upgrade_to_X
methods that it hasn't yet run, in order. The "currently deployed version" number is stored in a magic DeployedVersion Unit.
The upgrade_to_X
methods can choose to stay database-neutral and just use the (new) arena.add_property
, drop_property
, and rename_property
methods. But because each Schema is application-specific, you can also write optimized instructions for your known StorageManagers. For example, say you need to change an int property to a string. The "database-neutral" way would be to have additional Arena methods for such tasks. Some of those methods may be added in the future, but nothing's stopping you now from writing non-portable SQL statements if you know your app is only deployed on, say, Postgres (but you should probably assert that before you execute the SQL statements).
Anyway, I'd be interested to hear from anyone else who has written database-versioning tools. Save me from a pitfall if you can. Have fun with the new Schema class and let's see if there are a couple of other common methods (like
add_column
) that should go into the Arena and the StorageManagers.
After much woe, I think I finally tracked down the status problems I was having with modpython_gateway.py (which is now available on my "misc" Trac site). It should now correctly handle redirects, 404's, and .css and .js content. I think it also fixed my earlier "delayed content" problem.
I hereby nominate mod_python's status API for the "One Obvious Way To Do It" booby prize. Having req.status, a return value/status, and the option to raise a status makes far too many combinations.
The correct answer is: "nobody knows". But here are some ideas I've been kicking around the ol' cranium lately...
[09:32] *** now talking in #cherrypy [10:22] <Lawouach> where to start [10:22] <Lawouach> what's your basic idea toward 3.0? [10:22] <@fumanchu> oh, I have so many ;) [10:22] <Lawouach> lol [10:22] <Lawouach> say big general ones :) [10:22] <Lawouach> not details per se [10:23] <@fumanchu> 1) make CP have a kick-butt, non-CP-specific toolkit (lib/httptools), that is SO good that Quixote, Django, et al can't *help* but decide to use it instead of their own server processes [10:24] <@fumanchu> even if they don't like the way CP maps handlers to URL's, for example [10:24] <@fumanchu> they should be able to build a server with the behavior they like out of lib/httptools [10:25] <Lawouach> we want to be lib that rule them all :) [10:25] <@fumanchu> yup [10:26] <Lawouach> i agree as long as we don't become a framework on our own, but i already know it's not what you intend :) [10:26] <@fumanchu> right [10:26] <@fumanchu> it's an anti-framework approach [10:26] <@fumanchu> we make writing-a-web-framework into a weekend's work [10:27] <@fumanchu> take some from column A; try all of column B [10:27] <Lawouach> do you want to stay very low-level (aka HTTP wrapper level) or make it a bit higher level and provide functions such as the bast_match() we were talking about last week? [10:27] <@fumanchu> best_match would be fine as long as it doesn't depend upon cherrypy [10:28] <Lawouach> right, this was a bad example [10:28] <Lawouach> but basically where httptools should stop? [10:28] <@fumanchu> I think that can be open-ended [10:28] <Lawouach> i think we should keep the level you've been doing till now [10:29] <@fumanchu> 2) then, by pulling a ton of code out of _cphttptools (putting it in lib/httptools instead), I want to see if we can get the Request and Response objects down to a tiny size [10:34] <@fumanchu> the trunk version of _cphttptools is already 60% of its 2.1 size [10:35] <Lawouach> right. hmmm [10:37] <@fumanchu> and a *lot* of what's left is very OO [10:38] <@fumanchu> so, one idea I'm toying with: allow developers to use their own subclasses of Request and Response [10:40] <@fumanchu> if we make it super-easy to use custom Request subclasses, then they will want to start overriding Request.run [10:40] <@fumanchu> take out the filter logic, and Request.run becomes:
def _run(self, requestLine, headers, rfile):
self.headers = list(headers)
self.headerMap = httptools.HeaderMap()
self.simpleCookie = Cookie.SimpleCookie()
self.rfile = rfile
self.processRequestLine(requestLine)
try:
self.processHeaders()
self.processBody()
self.main()
cherrypy.response.finalize()
except cherrypy.RequestHandled:
pass
except (cherrypy.HTTPRedirect, cherrypy.HTTPError), inst:
inst.set_response()
cherrypy.response.finalize()
[10:40] <Lawouach> regarding the subclassing of request and response, i'm know that it could interest very much the guys behind itools [10:40] <@fumanchu> yes [10:40] <@fumanchu> and Ben Bangert (routes) [10:41] <@fumanchu> anyway, if Request.run is *that* simple, then who needs filters? [10:41] <@fumanchu> just code them procedurally into your Request.run method [10:43] <@fumanchu> looking over the filters that are built in... [10:44] <@fumanchu> I think that half could be done just as easily as lib/httptools functions [10:44] <@fumanchu> and half could be "always on" [10:44] <@fumanchu> (if we continue to improve them, like encodingfilter, to meet the HTP spec) [10:44] <@fumanchu> HTTP [10:44] <Lawouach> that's my white cheap :) (i don't think this expression exists so i make it up!) [10:45] <Lawouach> i really want CP to be HTTP conditionnaly compliant at least :) [10:45] <Lawouach> and maybe in CP 4.0 to be unconditionnaly compliant! [10:45] <Lawouach> :p [10:45] <@fumanchu> I completely agree [10:46] <@fumanchu> anyway, I want to stress that I'm still playing with these ideas [10:46] <@fumanchu> nothing's set in stone [10:47] <Lawouach> since you've be proposing them a while back, i've been a great fan of them [10:47] <@fumanchu> and trying to implement them will turn up lots of problems, I'm sure [10:47] <@fumanchu> oh, well thanks [10:47] <Lawouach> that's why i don't have so many different things to bring for cp 3.0 [10:51] <@fumanchu> one of the nice things about these ideas for 3.0 is that the bulk of the work can be done within the 2.x branch
Dear lazyweb,
After 6 hours, I am utterly stumped. I've got an application built with a popular Python web application server, via mod_python, and keep seeing data bleed from one request to the next. That is, if I:
OK
The requested URL /jjj.css was not found on this server.
Apache/2.0.55 (Win32) mod_ssl/2.0.55 OpenSSL/0.9.8a mod_python/3.2.2b Python/2.4.2 mod_auth_sspi/1.0.2 Server at skipper.amorhq.net Port 443 HTTP/1.1 404 Not Found Date: Tue, 22 Nov 2005 01:57:37 GMT Server: Apache/2.0.55 (Win32) mod_ssl/2.0.55 OpenSSL/0.9.8a mod_python/3.2.2b Python/2.4.2 mod_auth_sspi/1.0.2 Content-Length: 371 Keep-Alive: timeout=15, max=94 Connection: Keep-Alive Content-Type: text/html; charset=iso-8859-1Not Found
The requested URL /mmm.css was not found on this server.
Apache/2.0.55 (Win32) mod_ssl/2.0.55 OpenSSL/0.9.8a mod_python/3.2.2b Python/2.4.2 mod_auth_sspi/1.0.2 Server at skipper.amorhq.net Port 443
The body of request #2 is present in request #3, and so are the headers of request #3! Frightening.
This happens reliably with both Firefox and IE. It happens whether I use HTTPS or not. It happens whether I use authentication or not. It happens when I strip the modpython gateway-for-WSGI I wrote down to 80 lines.
It stops happening when I use CherryPy's builtin WSGI server, so I don't think any part of CP is to blame, which leaves a bug in mod_python or Apache2. I'm particularly inclined to blame them because, although CherryPy and Apache itself log both the missing responses as 404, Ethereal shows me that the actual third response, as received by the client, has a 200 response code!
So I'm stumped. Any solutions, pointers, or flights of debugging fantasy accepted.
Update Nov 6, 2005: Finally got it to work with Apache2-prefork on Unix (it only worked on mpm_winnt until now).
Update Oct 25, 2005: I was having a problem setting up a new install of my CherryPy application, using this recipe. It turned out that I didn't have the right interpreter_name in my PythonImport directive:
PythonImport module interpreter_name
Therefore, the CherryPy server started in a different intepreter than the one being used for the requests. It must exactly match the value of req.interpreter, and is case-sensitive. I've updated the code with comments to that effect (just to have it all in one place).
Update Aug 11, 2005: I was having a problem serving .css and .js pages. CherryPy's standalone WSGI server did fine, but mod_python did not. I finally tracked it down to the fact that I was both setting apache's req.status and returning req.status from the handler. Funky. It worked when I chose to simply return the value, and not set it.
Update June 5, 2005:
Pages take forever to terminate when returning a status of 200--apache.OK must be returned instead in that case.
Added code for hotshot profiling.
Added code for using paste.lint
As I mentioned I was doing last week, I wrote a more complete WSGI wrapper for modpython. Here it is. Feedback welcome. Phil Eby told me he'd like a mod_python wrapper for inclusion in wsgiref; he should feel free to use this one however he sees fit.
"""
WSGI wrapper for mod_python. Requires Python 2.2 or greater.
Example httpd.conf section for a CherryPy app called "mcontrol":
<Directory D:\htdocs\mcontrol>
SetHandler python-program
PythonHandler wsgiref.modpy_wrapper::handler
PythonOption application cherrypy.wsgiapp::wsgiApp
PythonOption import mcontrol.cherry::startup
</Directory>
"""
import sys
from mod_python import apache
from wsgiref.handlers import BaseCGIHandler
class InputWrapper(object):
def __init__(self, req):
self.req = req
def close(self):
pass
def read(self, size=-1):
return self.req.read(size)
def readline(self):
return self.req.readline()
def readlines(self, hint=-1):
return self.req.readlines(hint)
def __iter__(self):
line = self.readline()
while line:
yield line
# Notice this won't prefetch the next line; it only
# gets called if the generator is resumed.
line = self.readline()
class ErrorWrapper(object):
def __init__(self, req):
self.req = req
def flush(self):
pass
def write(self, msg):
self.req.log_error(msg)
def writelines(self, seq):
self.write(''.join(seq))
bad_value = ("You must provide a PythonOption '%s', either 'on' or 'off', "
"when running a version of mod_python < 3.1")
class Handler(BaseCGIHandler):
def __init__(self, req):
options = req.get_options()
# Threading and forking
try:
q = apache.mpm_query
except AttributeError:
threaded = options.get('multithread', '').lower()
if threaded == 'on':
threaded = True
elif threaded == 'off':
threaded = False
else:
raise ValueError(bad_value % "multithread")
forked = options.get('multiprocess', '').lower()
if forked == 'on':
forked = True
elif forked == 'off':
forked = False
else:
raise ValueError(bad_value % "multiprocess")
else:
threaded = q(apache.AP_MPMQ_IS_THREADED)
forked = q(apache.AP_MPMQ_IS_FORKED)
env = dict(apache.build_cgi_env(req))
if req.headers_in.has_key("authorization"):
env["HTTP_AUTHORIZATION"] = req.headers_in["authorization"]
BaseCGIHandler.__init__(self,
stdin=InputWrapper(req),
stdout=None,
stderr=ErrorWrapper(req),
environ=env,
multiprocess=forked,
multithread=threaded
)
self.request = req
self._write = req.write
def _flush(self):
pass
def send_headers(self):
self.cleanup_headers()
self.headers_sent = True
# Can't just return 200 or the page will hang until timeout
s = int(self.status[:3])
if s == 200:
self.finalstatus = apache.OK
else:
self.finalstatus = s
# the headers.Headers class doesn't have an iteritems method...
for key, val in self.headers.items():
if key.lower() == 'content-length':
if val is not None:
self.request.set_content_length(int(val))
elif key.lower() == 'content-type':
self.request.content_type = val
else:
self.request.headers_out[key] = val
_counter = 0
def profile(req):
# Call this function instead of handler
# to get profiling data for each call.
import hotshot, os.path
ppath = os.path.dirname(__file__)
if not os.path.exists(ppath):
os.makedirs(ppath)
global _counter
_counter += 1
ppath = os.path.join(ppath, "cp_%s.prof" % _counter)
prof = hotshot.Profile(ppath)
result = prof.runcall(handler, req)
prof.close()
return result
def handler(req):
config = req.get_config()
debug = int(config.get("PythonDebug", 0))
options = req.get_options()
# Because PythonImport cannot be specified per Directory or Location,
# take any 'import' PythonOption's and import them. If a function name
# in that module is provided (after the "::"), it will be called with
# the request as an argument. The module and function, if any, should
# be re-entrant (i.e., handle multiple threads), and, since they will
# be called per request, must be designed to run setup code only on the
# first request (a global 'first_request' flag is usually enough).
import_opt = options.get('import')
if import_opt:
atoms = import_opt.split('::', 1)
modname = atoms.pop(0)
module = __import__(modname, globals(), locals(), [''])
if atoms:
func = getattr(module, atoms[0])
func(req)
# Import the wsgi 'application' callable and pass it to Handler.run
modname, objname = options['application'].split('::', 1)
module = __import__(modname, globals(), locals(), [''])
app = getattr(module, objname)
h = Handler(req)
## from paste import lint
## app = lint.middleware(app)
h.run(app)
# finalstatus was set in Handler.send_headers()
return h.finalstatus