Optimal Syntax for Python Decorators

Introduction

An optimal decorator syntax

Like all programming languages, Python has many conventions. One of the strongest is that a series of statements will be evaluated linearly; that is, from top to bottom and left to right. Decorators are a distinct break from this linear imperative style and deserve a distinct syntax.

We argue that the current syntax, while quite distinct, is not the optimal syntax. We present an alternative syntax, and analyze its two most important characteristics: the use of an indented suite, and the preference of a keyword over punctuation for the required new token. We finish by recommending a keyword.

There are many details regarding the implementation of decorators which will not be addressed in this proposal, because they are orthogonal or extraneous to the core arguments. Details and issues which do not directly relate to the two proposed syntax changes are outside the scope of this document. A short list of extraneous issues includes:

  1. Order of execution of decorators: foo(bar(func)) versus bar(foo(func))
  2. Location of decorators before def (same for both syntaxes)
  3. Whether to allow multiple decorators per line
  4. Backwards portability
  5. The internals of decorator callables
  6. Beauty, Fear, Magic or Intuition
Any semantic or syntactic issue which is not addressed in this proposal retains the same syntax or semantic as in the pie-syntax implementation (current CVS, Aug 25, 2004).

Apologies to Guido: this is a dual-purpose document. It has two audiences and two goals, because time is short. The first audience is the Python community. However, because the second audience is you yourself, I apologize for the number of times I quote you and speak about you in the third person. ;)

A Persistent Alternative Syntax

The current syntax, "pie-before-def" 2, received strong opposition from portions of the Python community beginning with its inclusion in Python 2.4a2. An example of this syntax:

@funcattrs(status="experimental", author="BDFL")
@staticmethod
def longMethodNameForEffect(*args):
    """This method blah, blah."""
    raise NotYetImplemented

Dozens of syntax variants were proposed before and since that time; they are summarized on the Python decorator wiki 3. Recently, however, the syntax known as "J2" (or "def suite") has arisen as an option which the general Python community can support. An example of this syntax:

using:
    funcattrs(status="experimental", author="BDFL")
    staticmethod
def longMethodNameForEffect(*args):
    """This method blah, blah."""
    raise NotYetImplemented

We feel confident in proposing this syntax as a clear alternative to the existing syntax due to the following:

J2 is not a new syntax.

Shane Hathaway proposed this syntax at least as early as March 26, 2004 4. Guido himself included it in a list of only 3½ options in April:

"I'd like to have a bake-off, where we view a serious amount of code using decorators to the hilt with each of three proposed syntaxes:
...

3) Prefix suite (could use a different keyword than 'decorate')" 5
Unfortunately, no implementation was available for the J2 syntax at the time of the 'bake-off' request.

No part of J2 has been rejected outright by our BDFL.

Unlike most of the other syntax variants, J2 has consistently dodged the pronouncement bullet. There was a significant period of time in which Guido expressed his dislike for a new keyword; however, he has since stated more than once that a keyword might be allowed given a __future__ import:

"I really don't care if everybody thinks it [pie-syntax] is ugly. I do care to find out about usability issues... I also want to find out about superior syntax proposals (from __future__ import decorators might be acceptable)." 6
No substantive arguments against J2 have been found, as we show below.

Arguments

I. A suite is better than multiple @lines.

1. Rationale

One of the most powerful expectations present in imperative (state-based) languages is that the vast majority of code will be evaluated at runtime in the same order in which it appears on the page. Python is no exception. However, this is no way to write complete programs; conditional branching, loops, and subroutines have long provided alternative flows of control. That is, they cause run-time order to be different than compile-time order.

In Python, these "control flow" constructs (if, for, while, try, funcdef, and classdef) 7 all belong to the lexical category of "compound statements". They may utilize dependent "helper statements" (break, continue, raise, return and yield). These six compound constructs all use the colon as a delimiter to introduce a indented suite.

In the same way, decorators provide an alternative flow of control. They cause run-time order to differ from compile-time order. It is therefore natural and beneficial to adopt a decorator syntax which is similar to that of the existing flow-control constructs. We therefore recommend that decorators adopt a grammar which includes an indented suite.

There are four existing constructs which provide multiple suites: if/elif/else, for/else, while/else, and try/except/finally. All of these derive strong denotational weight from their indentation. Specifically, although the suites are indented, the multiple declarations themselves are not. This visually links the declarations, strengthening their relationship and dependency. None of the dependent suites (elif, else, except, or finally) "stand alone" because they would have no meaning if they did so. Outside of the functions they affect, decorators also lose both denotational and operational semantics. Indeed, if the multi-suite pattern is not followed, decorator referents are rendered ambiguous (in the sense that they do not follow any precedent). We therefore recommend that decorators also follow the multiple-suite pattern.

2. Objections

Objections to the use of an indented-suite syntax have been raised. One of the first was that "compound statements imply a nested namespace." Decorator-suites certainly would introduce a new kind of construct, the first construct wherein one dependent suite does not introduce a namespace (decorator), while the second suite does (def). However, two factors mitigate this issue. First, compound statements are currently ambivalent regarding namespaces; only two of the six introduce new namespaces (funcdef and classdef). Second, namespaces are wholly dependent upon control flow suites. The addition of a mixed-namespace construct can be seen as a refinement of the policy of suites in general, not a departure. As Guido put it when discussing thunks,

"Many users have only a vague notion of the difference between a syntactic construct and a function (witness repeated suggestions here and elsewhere that confuse these matters), yet nobody has a problem with understanding the difference in scope rules between 'def' and 'if' statements, for example." 8

The second major objection is that a suite for decorators violates the expectation "that a suite is a sequence of statements" 9. This is true; however, normal Python code shares that expectation. Therefore, the current pie-syntax shares this weakness. Further, it is almost exclusively within suites that one currently finds bare expressions which are not statements. For example, docstrings are most often found within function and class suites. Lambdas are another common example, albeit degenerate, of bare expressions in a suite-like structure (although lambdas do not officially possess 'suites' in the Grammar, they possess the delimiting colon, and are therefore denotational siblings of one-liner suites). Given these precedents, a suite is a more appropriate and consistent place (than normal linear code) in which to introduce a series of decorator expressions. Finally, although the decorator expressions appear in code to be declarative expressions, they behave at runtime like imperative statements.

Last, some have said the use of a suite is "too heavyweight" when used for a single decorator, the most common case. The use of a suite does use more characters (and more typing). We answer by pointing to exceptions as a precedent; it has been generally-accepted programming practice in the Python community for some time now to limit the first suite of a try/except block to a single statement ("narrow the exception"). Regardless, we find the extra whitespace a feature to embrace, rather than avoid (see below).

3. Additional benefits

Some additional benefits arise from the use of suites. First, as mentioned above, a suite strengthens the visual association between the decorators and the referent function. A paired, compound statement of <token>: and def: strengthens the fact that both are declarations; further, they each produce a new binding within the current scope. Note, in particular, that the result of a series of decorators is a single new binding. Pie-before-def syntax does not make this as clear as does a suite. Although the decorator suite does not introduce a new namespace, it may hint that some namespace trickery is going on--that any intermediate bindings are discarded.

Second, it is easier for humans to visually distinguish the location of the function signature with a decorator suite (when multiple decorators are present). It could be argued that another de facto expectation exists for Python: that flow-control suite headers are preceded by either a blank line or an associated, indented suite.

using:
    funcattrs(author="Guido van Rossum")
    memoize
    synchronize
    classmethod
def foo(cls, *args):
    pass
versus
@funcattrs(author="Guido van Rossum")
@memoize
@synchronize
@classmethod
def foo(cls, *args):
    pass
One drawback to the suite version is that the actual name of the function might become obscured, since most editors indent 4 spaces, causing the decorators, the function name, and the function code block to line up in the same starting column.

Third, a decorator suite strengthens the visual association of the decorators to each other, delimiting them both vertically (via one identifier for the whole set as opposed to one each) and horizontally (via shared indentation).

Fourth, source-code folding is easier with a suite.

Finally, the use of a suite for decorators provides greater flexibility for the future. Once decorators begin to be used in earnest, people will undoubtedly find new ways to use them, and will begin to want more power from decorator syntax 11. Some of these uses have already been put forward:

This proposal does not possess the scope or intent to detail every possible future for decorator semantics. However, the use of a suite for decorators gives a clear benefit to designing the syntax of those possible futures. Because suite-syntax uses existing syntax structures, other existing syntax can be more easily adapted to the decorator model in future releases of Python. The suite provides all of the semantic power of statements, without having to tag each decorator with its own 'decorator token'. Perhaps most powerfully of all, decorator suites could someday allow a mixture of decorators and normal Python statements within the same suite, without consequently making the referent of each decorator ambiguous.

II. A keyword is better than punctuation for a new token.

The choice whether to use a suite for decorator statements is an important one. Equally as important is the choice between punctuation or keyword for the required new syntax.

1. No new token category

First, browsing through the Language Reference, one is immediately struck by the simplicity of the lexicon. Besides whitespace, tokens are either identifiers, keywords, literals, operators, or delimiters. The @-prefix for decorators is unfortunately none of these. Prefix punctuation appears in only two other places in standard Python: *args and **kwargs, and the "reserved classes of identifiers", those with leading underscores 12. Pie-syntax is certainly not a reserved class of identifiers, and mysteriously, *args and **kwargs are not mentioned in the section on lexical analysis at all. In short, prefix punctuation for decorators requires the development of an entirely new category of token. The use of a keyword, by contrast, uses existing syntax to provide a new token without a new category.

2. Matches existing use of tokens

Second, keywords have traditionally been used to express control flow in Python. All of the existing flow-control statements, both compound and simple, use keywords. As the authors of PEP 255 (Simple Generators) put the matter when discussing 'yield':

Q. Why a new keyword for "yield"? Why not a builtin function instead?
A. Control flow is much better expressed via keyword in Python, and yield is a control construct. 13
Punctuation, by contrast, is used almost exclusively for operators and delimiters. The use of punctuation in the context of decorators would, we believe, break these long-standing conventions and harm readability 14. In addition, it would restrict future uses of a limited set of punctuation:
"On the plus side, Java's @name(kwargs) syntax allows us to put decorators in front methods and classes without ambiguous syntax; on the minus side, using up a potential operator character for one specific purpose should not be done lightly." 15
In other words, we are in far more danger of running out of punctuation (and therefore, new tokens for operators and delimiters) than we are in danger of running out of keywords (and therefore, new tokens for control flow constructs).

3. Enforces necessary change control

Third, new punctuation does not enforce code-level structure to manage the change process. At least two developers (of Leo and IPython 16) have already come forward, stating that their tools will have to be modified to accomodate the proposed @ symbol for decorators. The unfortunate reality of large-scale open-source development is that we cannot know in advance how many more such cases exist; without code-level change control, we are helpless to mitigate the risk. A new keyword, however, would be required to take advantage of a __future__ declaration 17, which provides an established, well-known, successful mechanism for change control.

In particular, __future__ provides a more measured release schedule. During the interim (when "from __future__ import decorators" is required to use decorators), developers gain the ability to apply the new syntax on an "as-needed" basis; that is, they can use the new syntax in one application while delaying the syntax in another. Where __future__ is not invoked, the keyword will continue to function as a normal identifier. For example, one developer may have immediate need for decorators, while another may need a substantial amount of time to migrate their code. Admittedly, the latter would only face issues under the pie-syntax if using Leo, IPython, or some other nonstandard use of @. Again, however, we are not aware of these other uses, whether of @ or a keyword token. The use of __future__ mitigates this risk.

Finally, the use of __future__ provides code-level documentation of the change process. From PEP 236:

"To document when incompatible changes were introduced, and when they will be--or were--made mandatory. This is a form of executable documentation, and can be inspected programatically via importing __future__ and examining its contents." 17

4. Python decorators are not Java annotations nor .Net attributes

Python decorators have been through several transformations. Most recently, decorators adopted syntax from C#, and then, Java. In particular, this similarity of syntax has been put forward as a benefit in favor of these syntaxes; specifically, that users of these other languages would have an easier time deciphering Python decorators, based upon their familiarity with the syntax in the other language(s).

The current @-decorator syntax is inspired by Java's "annotations". In fact, the authors of the Java annotation feature state that the @-symbol is a mnemonic for annotation type. However, Python decorators differ significantly from Java's use of that syntax. Most importantly, "annotations" are limited to attributes and other non-transformative operations in Java. From the JDK 1.5 specification:

"This facility allows developers to define custom annotation types and to annotate fields, methods, classes, and other program elements with annotations corresponding to these types. These annotations do not directly affect the semantics of a program." 18 (emphasis added)
Java annotations provide semantic transparency; the referent is not modified. One may use "interceptor" functions like @trace, but the annotated function is not wrapped. In Java, function transformations continue to be provided by keywords such as static.

In contrast, all Python decorators accept one function and return one function. Many of them will return a new function object. Many will change function signatures, affecting both the decorated function and its callers. The most common decorators, classmethod and staticmethod, are among these. In short, Python decorators propose the opposite implementation of Java's annotations.

Similarly, the previous []-before-def syntax was inspired by Microsoft's C# language. But C# also limits the expressivity of what it calls "attributes". They are similar to Python decorators in the sense that both involve what appear to be expressions as opposed to statements; however, .Net attributes really are non-statements; they do not modify their referents so much as attach themselves to them. Microsoft often calls attributes "declarative" as opposed to "imperative" 19. Further, they are required to be commutative 20, a requirement which would cripple many uses of Python decorators 21.

Both syntaxes (those inspired by C# and Java) suffer doubly: although they promise semantic parallelism, they not only fail to deliver it, but deliver the opposite semantic. A keyword syntax is free to define its own semantics from scratch, and will lessen the confusion of developers who are new to Python, yet experienced in these other languages.

III. Choosing a keyword

If a keyword is to be chosen over @ or other punctuation, the question remains, "which word should it be?" So many words have been proposed that it would be difficult to examine each one in turn. Therefore, we begin by establishing guidelines for the selection of a keyword. Such a keyword:

  1. Should not be used widely or critically as an identifier in existing Python code.
  2. Should be easy to remember when writing new code.
  3. Should be easy to remember when reading existing code.
  4. Should be easy to search for, in both docs and Google.
  5. Should not be a word with a planned future of different semantics. This rules out "as", and probably "with". 22
  6. Should not be "def", which Guido rejected as "too weird. The stutter doesn't have any semantic connotation to it and is bound to confuse source-scanning tools." 23
  7. Should either pair strongly with "def", or at least not stand alone.
  8. Should not address only one aspect of decorators.
  9. Should not be a form of the word "decorate". The term "decorate" conflicts with two separate concepts: both the GoF Decorator pattern (which is a runtime wrapper, not a compile-time one), and with our own beloved "decorate-sort-undecorate" pattern (aka Schwartzian or Guttman-Rosler Transform).

Candidates for keywords have fallen into several camps. Many of the more common candidates suffer by addressing only one aspect of decorators. Prime examples are keywords such as "transform" and "wrap", which refer to the nature of those decorators which return new function objects; such keywords do not fare well when applied to decorators which merely annotate. Conversely, the words "declare", "decorate", and "decor" address the annotative aspects of decorators, ignoring the transformative aspects. In addition, such words would tend to clash with the names given to decorators themselves; for example, the test suite for decorators names a decorator function "decorate". It isn't difficult to imagine users wanting decorators named "transform" and "wrap", as well, even if such generic names are discouraged.

As discussed eariler, decorators have no meaning apart from their referent functions. Indeed, this is a strong part of the case for a suite syntax, which more strongly denotes the dependency. We recommend that a keyword be chosen which directly refers to this dependency. Some examples:

as, by, confer, having, helper, meta, per, predef, through, using, via
Of these, the words "using", "as", "per", and "predef" have each had multiple advocates within the community.

Looking again at existing control flow constructs, we notice that keyword choice follows purpose. "Try" is a verb and its object is the suite. "Def(ine)" is a verb, and its object is the suite 24. "If", "while", and "for" all refer to their declarations or tests, not their suites, and function adverbially. For example, in the statement "while (x == 3)", the object of the keyword "while" is "x == 3", not the suite of statements which follow. More importantly, the dependent clauses (else, except, and finally) all function adverbially. Since decorator suites are also dependent and also function adverbially, the keyword for decorator suites should either be an adverb, or a preposition or conjunction which is often used adverbially.

Of the candidates, we recommend the keyword "using", because:

Conclusion

We began by stating that a distinct new language feature deserves a distinct syntax, and pie-syntax is certainly distinct! We have shown that alternatives exist, particularly, that the use of a suite, and the keyword "using" for the new token, supply a new, distinct construct which better fits the precedents of existing Python. In the absence of further technical arguments, the "J2" syntax results in decorators which are more conventional and readable, yet also more extensible and manageable. In contrast, the current "A1" syntax is more radical and also more limited. We leave the decision between these outcomes to our trusty BDFL.

Signatories

If you wish to add your name to this proposal, either for, against, or abstaining, please send mail to either comp.lang.python or the author (fumanchu@amor.org). If you reject the proposal, please include a summary of your objections so that your opinion can be included toward Guido's more complete understanding.

In favor, stating that the proposal as written provides a more acceptable syntax than the current syntax, and that part or all of the Arguments should be implemented,

Robert E. Brewer
Michael Sparks
Paul McGuire

Aahz
Alex Martelli
Andrew Durdin
Antonio B. Leal
Arien Malec
Austin Luminais
Bjørn Ådlandsvik
Chris Ryland
Christopher T. King
Colin J. Williams
Dan Christensen
Dan Pierson
Dan Schmidt
Daniel Arbuckle
Daniel Bickett
Darragh Sherwin
Dave Brueck
David Fraser
David Goodger
David Mertz, Ph.D.
David Pokorny
David Vaughan
Dennis Benzinger
Detlef Lannert
Don Welch
Douglas McNeil
eltronic
Facundo Batista
Fernando Perez
Francesco Bochicchio
François Pinard
Dr. Gary Herron
Graham Fawcett
Gyro Funch
Jamie Radcliffe
JanC
Jeff Shannon
Jess Austin
Jim J Jewett
John Carter
John Crichton
John Ehresman
John Keeling
John Marshall
Jon Nicoll
Ksenia Marasanova
Leif K-Brooks
Mahesh Padmanabhan
Marc Boeren
Mark Bassett
Matteo Dell'Amico
Michele Simionato
Nick Coghlan
Nick Efford
Nick Jacobson
Nicolas Fleury
Nigel Rowe
Paul L. Du Bois
Paul Paterson
Peter Hansen
Peter Mayne
Peter Otten
Pierre Frédéric Caillaud
Richard Jones
Richard Kuhns
Roman Suzi
Russell E Owen
Russell Salsbury
Scott David Daniels
Simon Brunning
Steven Bethard
Terry J. Reedy
Tim Hochberg
Tim Hoffman
Troy Melhase
Ville Vainio
Walter Dörwald

Against, stating that the existing syntax is to be preferred over the syntax or Arguments presented herein,

Arthur: Pie-syntax better expresses the intent of a programmer shortcut.
Bernhard Herzog: Not convincing. Pie-syntax reads better. "using" is not a good choice of keyword.
Cristophe Cavalaria: Pie-syntax reads better. "using" not a good choice of keyword.
Greg Ewing
Paul Moore: Not convincing. Pie-syntax already in place.
Thomas Heller: Pie-syntax reads better.

Abstaining, stating that the choice is either undecidable or irrelevant,

Ganesan Rajagopal: not convinced decorators are a good thing for Python.
Neil Hodgson
Uriel M.: Decorators should stay out of the language until there is a better syntax.

Appendix

Code

The current patch has been provided by Michael Sparks, and is available at http://www.python.org/sf/1013835. The previous minor scoping issue 25 was fixed on Aug 21, 2004.

Grammar

pie syntax (CVS, Aug 17 2004):

    decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
    decorators: decorator+
    funcdef: [decorators] 'def' NAME parameters ':' suite

suite-and-keyword syntax:

    decorator: dotted_name [ '(' [arglist] ')' ] NEWLINE
    decorators: decorator+
    funcdef: ['using' ':' (decorator |NEWLINE INDENT decorators DEDENT) ]
             'def' NAME parameters ':'suite

Notes

[1] http://mail.python.org/pipermail/python-dev/2004-March/043546.html
[2] Option A1 on the Python Decorator Wiki.
[3] http://www.python.org/moin/PythonDecorators
[4] http://mail.python.org/pipermail/python-dev/2004-March/043541.html
[5] http://mail.python.org/pipermail/python-dev/2004-April/043884.html
[6] http://mail.python.org/pipermail/python-dev/2004-August/046878.html
[7] "import" and "from" have been left out of the discussion for clarity.
[8] http://mail.python.org/pipermail/python-dev/2003-February/032712.html
[9] GvR, http://article.gmane.org/gmane.comp.python.devel/58961
[10] http://mail.python.org/pipermail/python-dev/2004-August/047131.html
[11] GvR, http://mail.python.org/pipermail/python-dev/2004-March/043745.html
"1) If we do decorators at all, decorators should be allowed to be arbitrary expressions.

2) Since we allow arbitrary expressions, decorators provide a much better way to set function attributes than the current way."
See also: http://mail.python.org/pipermail/python-dev/2004-April/043957.html
[12] http://docs.python.org/ref/id-classes.html
[13] http://www.python.org/peps/pep-0255.html
[14] GvR, http://mail.python.org/pipermail/python-dev/2003-January/032649.html
"I've got a little bit of philosophy on the use of keywords vs. punctuation. While punctuation is concise, and often traditional for things like arithmetic operations, giving punctuation too much power can lead to loss of readability. (So yes, I regret significant trailing commas in print and tuples -- just a little bit.)"

[15] GvR, http://mail.python.org/pipermail/python-dev/2004-June/045125.html
[16] Leo: http://mail.python.org/pipermail/python-list/2004-August/232143.html
IPython: http://mail.python.org/pipermail/python-dev/2004-August/047481.html
[17] See PEP 236, Back to the __future__. http://www.python.org/peps/pep-0236.html
[18] http://www.jcp.org/en/jsr/detail?id=175
[19] http://msdn.microsoft.com/library/en-us/cpguide/html/cpcondemands.asp
[20] http://msdn.microsoft.com/library/en-us/csspec/html/vclrfcsharpspec_17_2.asp
[21] http://article.gmane.org/gmane.comp.python.devel/58961
[22] Msg 12 and ff: http://groups.google.com/groups?th=1c239fdf6a989b11&seekm=mailman.2225.1093282717.5135.python-list%40python.org#link12
[23] http://mail.python.org/pipermail/python-dev/2004-April/043933.html
[24] Truly, the object of "def" is its suite. It implies the phrase, "define suite as name," where "as name" is a dependent clause.
[25] http://mail.python.org/pipermail/python-list/2004-August/233591.html