The Hand of FuManChu - Latest comments on Python concurrency syntax

In response to: Python concurrency syntax

admin [Member] — Tue, 05 Jun 2007 23:52:57 +0000

Right; avoiding "non-thread-safe" collections is the purpose of the idea, and class/module/instance dicts are collections. Proposing that classes and modules be immutable is an attempt to maximize the volume of code that can be shared.

And, yes, that would mean you can't say "MyClass.foo = something" outside of the class, at least not without an explicit "monkey" declaration either by the caller (user) or the callee (author) to mean, "this attribute is now thread-global". That seems a horrible thing for a user to be able to do, until you realize they already do it (see the os.rmdir example above).

In response to: Python concurrency syntax

mike bayer [Visitor] — Tue, 05 Jun 2007 23:14:40 +0000

oh. why not have dict on classes/modules be marked as "shareable"? ? other than you've now opened up a "non-thread-safe" collection....or is that the reason ?

also what does it really mean for class.dict to be immutable...does it mean I cant say MyClass.foo = something outside of the "class" declaration itself ?

In response to: Python concurrency syntax

admin [Member] — Tue, 05 Jun 2007 22:44:40 +0000

Hi Mike,

In my very humble musing, yes, each thread would get its own "x", but only because the "object" to which the name "x" refers is a mutable object. If "x" referred to an immutable, all threads could share it without any further intervention. This is admittedly tricky with Python's dynamic typing, but I think this automatic creation of threadlocals would only have to occur for globals and object attributes (and possibly cell references), not locals, and not function arguments.

I would imagine the copy would occur on the first get/set/del operation, just as it does for the current threading.local implementation.

In this scheme, if you don't make dict on classes/modules immutable, you end up localizing almost everything; you might as well go with multiple processes at that point. Maybe I should run through Dejavu (and perhaps CherryPy) and put some numbers to these vague intuitions of mine... ;)

In response to: Python concurrency syntax

mike bayer [Visitor] — Tue, 05 Jun 2007 21:59:04 +0000

Hi Robert -

can you expound upon what the "shared" syntax would look like? suppose I declare "x=[]" at the top of my module. Now I use the "thread" module to run two threads, each using a worker function that randomly appends values to "x". How does the "shared" syntax prevent that list from being modified by both threads? does "x" create copies of itself local to each thread when it detects a modify operation ? how does making dict on classes/modules immutable have anything to do with that (unless you mean, no more module-level globals or class-level variables )?

In response to: Python concurrency syntax

admin [Member] — Tue, 05 Jun 2007 21:54:14 +0000

Collin, could that not be solved with cooperative declaration between authors and users? You're right; allowing authors the sole power to @share would be to inter-thread sharing what Java's "private" modifier is to inter-class sharing--certainly not Pythonic. But in the cases where I've monkeypatched someone else's module, I would not have minded adding an explicit declaration to do so--in fact, I can see that being a beneficial construct for understanding and documentation:

import anotherlib

with shared(anotherlib):
anotherlib.os.rmdir = myrmdir

...especially if the name is fairly easy to grep for. Maybe we could name it "monkey" instead of "shared"? ;)

In response to: Python concurrency syntax

admin [Member] — Tue, 05 Jun 2007 21:06:05 +0000

Doug, I realize Python is already used widely for large systems. I simply mean that when concurrency-aware code is mentioned, Erlang et al are mentioned far more often than Python.

In response to: Python concurrency syntax

Collin Winter [Visitor] — Tue, 05 Jun 2007 21:03:35 +0000

Any of the "make such-and-such immutable" ideas would greatly limit what I regard to be Python's greatest strength: testability. Currently I can manipulate a module's namespace to influence how it perceives (say) the os module, perhaps replacing rmdir() with a function that will test an otherwise-hard-to-test code path in my library.

I wish people would just accept that threading is not one of Python's strengths and move on -- or better, write a decent interprocess queue so that MP code is easier.

In response to: Python concurrency syntax

Doug Napoleone [Visitor] — Tue, 05 Jun 2007 21:02:12 +0000

Question: What makes you think Python is not already big in the "large systems" space?
There are numerous examples of it already being there (Google, YouTube, SoE, DreamWorks, VMWare, EVE, etc, etc, etc, etc).

I do believe that a strict 'thread private by default' system would be fantastic for python. My only concern would be for those of us who like python for prototyping, and implement in other languages. The GIL can be a stumbling block already, and such a change would be a death blow to prototyping threaded systems for implementation in C++.