Like all programming languages, Python has many conventions. One of the strongest is that a series of statements will be evaluated linearly; that is, from top to bottom and left to right. Decorators are a distinct break from this linear imperative style and deserve a distinct syntax.
We argue that the current syntax, while quite distinct, is not the optimal syntax. We present an alternative syntax, and analyze its two most important characteristics: the use of an indented suite, and the preference of a keyword over punctuation for the required new token. We finish by recommending a keyword.
There are many details regarding the implementation of decorators which will not be addressed in this proposal, because they are orthogonal or extraneous to the core arguments. Details and issues which do not directly relate to the two proposed syntax changes are outside the scope of this document. A short list of extraneous issues includes:
Apologies to Guido: this is a dual-purpose document. It has two audiences and two goals, because time is short. The first audience is the Python community. However, because the second audience is you yourself, I apologize for the number of times I quote you and speak about you in the third person. ;)
The current syntax, "pie-before-def" 2, received strong opposition from portions of the Python community beginning with its inclusion in Python 2.4a2. An example of this syntax:
@funcattrs(status="experimental", author="BDFL")
@staticmethod
def longMethodNameForEffect(*args):
"""This method blah, blah."""
raise NotYetImplemented
Dozens of syntax variants were proposed before and since that time; they are summarized on the Python decorator wiki 3. Recently, however, the syntax known as "J2" (or "def suite") has arisen as an option which the general Python community can support. An example of this syntax:
using:
funcattrs(status="experimental", author="BDFL")
staticmethod
def longMethodNameForEffect(*args):
"""This method blah, blah."""
raise NotYetImplemented
We feel confident in proposing this syntax as a clear alternative to the existing syntax due to the following:
Shane Hathaway proposed this syntax at least as early as March 26, 2004 4. Guido himself included it in a list of only 3½ options in April:
"I'd like to have a bake-off, where we view a serious amount of code using decorators to the hilt with each of three proposed syntaxes:Unfortunately, no implementation was available for the J2 syntax at the time of the 'bake-off' request.
...
3) Prefix suite (could use a different keyword than 'decorate')" 5
Unlike most of the other syntax variants, J2 has consistently dodged the pronouncement bullet. There was a significant period of time in which Guido expressed his dislike for a new keyword; however, he has since stated more than once that a keyword might be allowed given a __future__ import:
"I really don't care if everybody thinks it [pie-syntax] is ugly. I do care to find out about usability issues... I also want to find out about superior syntax proposals (from __future__ import decorators might be acceptable)." 6No substantive arguments against J2 have been found, as we show below.
One of the most powerful expectations present in imperative (state-based) languages is that the vast majority of code will be evaluated at runtime in the same order in which it appears on the page. Python is no exception. However, this is no way to write complete programs; conditional branching, loops, and subroutines have long provided alternative flows of control. That is, they cause run-time order to be different than compile-time order.
In Python, these "control flow" constructs (if, for, while, try, funcdef, and classdef) 7 all belong to the lexical category of "compound statements". They may utilize dependent "helper statements" (break, continue, raise, return and yield). These six compound constructs all use the colon as a delimiter to introduce a indented suite.
In the same way, decorators provide an alternative flow of control. They cause run-time order to differ from compile-time order. It is therefore natural and beneficial to adopt a decorator syntax which is similar to that of the existing flow-control constructs. We therefore recommend that decorators adopt a grammar which includes an indented suite.
There are four existing constructs which provide multiple suites: if/elif/else, for/else, while/else, and try/except/finally. All of these derive strong denotational weight from their indentation. Specifically, although the suites are indented, the multiple declarations themselves are not. This visually links the declarations, strengthening their relationship and dependency. None of the dependent suites (elif, else, except, or finally) "stand alone" because they would have no meaning if they did so. Outside of the functions they affect, decorators also lose both denotational and operational semantics. Indeed, if the multi-suite pattern is not followed, decorator referents are rendered ambiguous (in the sense that they do not follow any precedent). We therefore recommend that decorators also follow the multiple-suite pattern.
Objections to the use of an indented-suite syntax have been raised. One of the first was that "compound statements imply a nested namespace." Decorator-suites certainly would introduce a new kind of construct, the first construct wherein one dependent suite does not introduce a namespace (decorator), while the second suite does (def). However, two factors mitigate this issue. First, compound statements are currently ambivalent regarding namespaces; only two of the six introduce new namespaces (funcdef and classdef). Second, namespaces are wholly dependent upon control flow suites. The addition of a mixed-namespace construct can be seen as a refinement of the policy of suites in general, not a departure. As Guido put it when discussing thunks,
"Many users have only a vague notion of the difference between a syntactic construct and a function (witness repeated suggestions here and elsewhere that confuse these matters), yet nobody has a problem with understanding the difference in scope rules between 'def' and 'if' statements, for example." 8
The second major objection is that a suite for decorators violates the expectation "that a suite is a sequence of statements" 9. This is true; however, normal Python code shares that expectation. Therefore, the current pie-syntax shares this weakness. Further, it is almost exclusively within suites that one currently finds bare expressions which are not statements. For example, docstrings are most often found within function and class suites. Lambdas are another common example, albeit degenerate, of bare expressions in a suite-like structure (although lambdas do not officially possess 'suites' in the Grammar, they possess the delimiting colon, and are therefore denotational siblings of one-liner suites). Given these precedents, a suite is a more appropriate and consistent place (than normal linear code) in which to introduce a series of decorator expressions. Finally, although the decorator expressions appear in code to be declarative expressions, they behave at runtime like imperative statements.
Last, some have said the use of a suite is "too heavyweight" when used for a single decorator, the most common case. The use of a suite does use more characters (and more typing). We answer by pointing to exceptions as a precedent; it has been generally-accepted programming practice in the Python community for some time now to limit the first suite of a try/except block to a single statement ("narrow the exception"). Regardless, we find the extra whitespace a feature to embrace, rather than avoid (see below).
Some additional benefits arise from the use of suites. First, as mentioned above, a suite strengthens the visual association between the decorators and the referent function. A paired, compound statement of <token>: and def: strengthens the fact that both are declarations; further, they each produce a new binding within the current scope. Note, in particular, that the result of a series of decorators is a single new binding. Pie-before-def syntax does not make this as clear as does a suite. Although the decorator suite does not introduce a new namespace, it may hint that some namespace trickery is going on--that any intermediate bindings are discarded.
Second, it is easier for humans to visually distinguish the location of the function signature with a decorator suite (when multiple decorators are present). It could be argued that another de facto expectation exists for Python: that flow-control suite headers are preceded by either a blank line or an associated, indented suite.
using:
funcattrs(author="Guido van Rossum")
memoize
synchronize
classmethod
def foo(cls, *args):
pass
versus
@funcattrs(author="Guido van Rossum")
@memoize
@synchronize
@classmethod
def foo(cls, *args):
pass
One drawback to the suite version is that the actual name of the
function might become obscured, since most editors indent 4 spaces,
causing the decorators, the function name, and the function code
block to line up in the same starting column.
Third, a decorator suite strengthens the visual association of the decorators to each other, delimiting them both vertically (via one identifier for the whole set as opposed to one each) and horizontally (via shared indentation).
Fourth, source-code folding is easier with a suite.
Finally, the use of a suite for decorators provides greater flexibility for the future. Once decorators begin to be used in earnest, people will undoubtedly find new ways to use them, and will begin to want more power from decorator syntax 11. Some of these uses have already been put forward:
This proposal does not possess the scope or intent to detail every possible future for decorator semantics. However, the use of a suite for decorators gives a clear benefit to designing the syntax of those possible futures. Because suite-syntax uses existing syntax structures, other existing syntax can be more easily adapted to the decorator model in future releases of Python. The suite provides all of the semantic power of statements, without having to tag each decorator with its own 'decorator token'. Perhaps most powerfully of all, decorator suites could someday allow a mixture of decorators and normal Python statements within the same suite, without consequently making the referent of each decorator ambiguous.
The choice whether to use a suite for decorator statements is an important one. Equally as important is the choice between punctuation or keyword for the required new syntax.
First, browsing through the Language Reference, one is immediately struck by the simplicity of the lexicon. Besides whitespace, tokens are either identifiers, keywords, literals, operators, or delimiters. The @-prefix for decorators is unfortunately none of these. Prefix punctuation appears in only two other places in standard Python: *args and **kwargs, and the "reserved classes of identifiers", those with leading underscores 12. Pie-syntax is certainly not a reserved class of identifiers, and mysteriously, *args and **kwargs are not mentioned in the section on lexical analysis at all. In short, prefix punctuation for decorators requires the development of an entirely new category of token. The use of a keyword, by contrast, uses existing syntax to provide a new token without a new category.
Second, keywords have traditionally been used to express control flow in Python. All of the existing flow-control statements, both compound and simple, use keywords. As the authors of PEP 255 (Simple Generators) put the matter when discussing 'yield':
Q. Why a new keyword for "yield"? Why not a builtin function instead?Punctuation, by contrast, is used almost exclusively for operators and delimiters. The use of punctuation in the context of decorators would, we believe, break these long-standing conventions and harm readability 14. In addition, it would restrict future uses of a limited set of punctuation:
A. Control flow is much better expressed via keyword in Python, and yield is a control construct. 13
"On the plus side, Java's @name(kwargs) syntax allows us to put decorators in front methods and classes without ambiguous syntax; on the minus side, using up a potential operator character for one specific purpose should not be done lightly." 15In other words, we are in far more danger of running out of punctuation (and therefore, new tokens for operators and delimiters) than we are in danger of running out of keywords (and therefore, new tokens for control flow constructs).
Third, new punctuation does not enforce code-level structure to manage the change process. At least two developers (of Leo and IPython 16) have already come forward, stating that their tools will have to be modified to accomodate the proposed @ symbol for decorators. The unfortunate reality of large-scale open-source development is that we cannot know in advance how many more such cases exist; without code-level change control, we are helpless to mitigate the risk. A new keyword, however, would be required to take advantage of a __future__ declaration 17, which provides an established, well-known, successful mechanism for change control.
In particular, __future__ provides a more measured release schedule. During the interim (when "from __future__ import decorators" is required to use decorators), developers gain the ability to apply the new syntax on an "as-needed" basis; that is, they can use the new syntax in one application while delaying the syntax in another. Where __future__ is not invoked, the keyword will continue to function as a normal identifier. For example, one developer may have immediate need for decorators, while another may need a substantial amount of time to migrate their code. Admittedly, the latter would only face issues under the pie-syntax if using Leo, IPython, or some other nonstandard use of @. Again, however, we are not aware of these other uses, whether of @ or a keyword token. The use of __future__ mitigates this risk.
Finally, the use of __future__ provides code-level documentation of the change process. From PEP 236:
"To document when incompatible changes were introduced, and when they will be--or were--made mandatory. This is a form of executable documentation, and can be inspected programatically via importing __future__ and examining its contents." 17
Python decorators have been through several transformations. Most recently, decorators adopted syntax from C#, and then, Java. In particular, this similarity of syntax has been put forward as a benefit in favor of these syntaxes; specifically, that users of these other languages would have an easier time deciphering Python decorators, based upon their familiarity with the syntax in the other language(s).
The current @-decorator syntax is inspired by Java's "annotations". In fact, the authors of the Java annotation feature state that the @-symbol is a mnemonic for annotation type. However, Python decorators differ significantly from Java's use of that syntax. Most importantly, "annotations" are limited to attributes and other non-transformative operations in Java. From the JDK 1.5 specification:
"This facility allows developers to define custom annotation types and to annotate fields, methods, classes, and other program elements with annotations corresponding to these types. These annotations do not directly affect the semantics of a program." 18 (emphasis added)Java annotations provide semantic transparency; the referent is not modified. One may use "interceptor" functions like @trace, but the annotated function is not wrapped. In Java, function transformations continue to be provided by keywords such as static.
In contrast, all Python decorators accept one function and return one function. Many of them will return a new function object. Many will change function signatures, affecting both the decorated function and its callers. The most common decorators, classmethod and staticmethod, are among these. In short, Python decorators propose the opposite implementation of Java's annotations.
Similarly, the previous []-before-def syntax was inspired by Microsoft's C# language. But C# also limits the expressivity of what it calls "attributes". They are similar to Python decorators in the sense that both involve what appear to be expressions as opposed to statements; however, .Net attributes really are non-statements; they do not modify their referents so much as attach themselves to them. Microsoft often calls attributes "declarative" as opposed to "imperative" 19. Further, they are required to be commutative 20, a requirement which would cripple many uses of Python decorators 21.
Both syntaxes (those inspired by C# and Java) suffer doubly: although they promise semantic parallelism, they not only fail to deliver it, but deliver the opposite semantic. A keyword syntax is free to define its own semantics from scratch, and will lessen the confusion of developers who are new to Python, yet experienced in these other languages.
If a keyword is to be chosen over @ or other punctuation, the question remains, "which word should it be?" So many words have been proposed that it would be difficult to examine each one in turn. Therefore, we begin by establishing guidelines for the selection of a keyword. Such a keyword:
Candidates for keywords have fallen into several camps. Many of the more common candidates suffer by addressing only one aspect of decorators. Prime examples are keywords such as "transform" and "wrap", which refer to the nature of those decorators which return new function objects; such keywords do not fare well when applied to decorators which merely annotate. Conversely, the words "declare", "decorate", and "decor" address the annotative aspects of decorators, ignoring the transformative aspects. In addition, such words would tend to clash with the names given to decorators themselves; for example, the test suite for decorators names a decorator function "decorate". It isn't difficult to imagine users wanting decorators named "transform" and "wrap", as well, even if such generic names are discouraged.
As discussed eariler, decorators have no meaning apart from their referent functions. Indeed, this is a strong part of the case for a suite syntax, which more strongly denotes the dependency. We recommend that a keyword be chosen which directly refers to this dependency. Some examples:
as, by, confer, having, helper, meta, per, predef, through, using, viaOf these, the words "using", "as", "per", and "predef" have each had multiple advocates within the community.
Looking again at existing control flow constructs, we notice that keyword choice follows purpose. "Try" is a verb and its object is the suite. "Def(ine)" is a verb, and its object is the suite 24. "If", "while", and "for" all refer to their declarations or tests, not their suites, and function adverbially. For example, in the statement "while (x == 3)", the object of the keyword "while" is "x == 3", not the suite of statements which follow. More importantly, the dependent clauses (else, except, and finally) all function adverbially. Since decorator suites are also dependent and also function adverbially, the keyword for decorator suites should either be an adverb, or a preposition or conjunction which is often used adverbially.
Of the candidates, we recommend the keyword "using", because:
using:
classmethod
funcattrs(author="Guido van Rossum")
def foo(cls):
pass
reads: "using classmethod, funcattrs(...), def[ine] foo..."We began by stating that a distinct new language feature deserves a distinct syntax, and pie-syntax is certainly distinct! We have shown that alternatives exist, particularly, that the use of a suite, and the keyword "using" for the new token, supply a new, distinct construct which better fits the precedents of existing Python. In the absence of further technical arguments, the "J2" syntax results in decorators which are more conventional and readable, yet also more extensible and manageable. In contrast, the current "A1" syntax is more radical and also more limited. We leave the decision between these outcomes to our trusty BDFL.
If you wish to add your name to this proposal, either for, against, or abstaining, please send mail to either comp.lang.python or the author (fumanchu@amor.org). If you reject the proposal, please include a summary of your objections so that your opinion can be included toward Guido's more complete understanding.
Arthur: Pie-syntax better expresses the intent of a programmer shortcut.
Bernhard Herzog: Not convincing. Pie-syntax reads better. "using" is not a good choice of keyword.
Cristophe Cavalaria: Pie-syntax reads better. "using" not a good choice of keyword.
Greg Ewing
Paul Moore: Not convincing. Pie-syntax already in place.
Thomas Heller: Pie-syntax reads better.
Ganesan Rajagopal: not convinced decorators are a good thing for Python.
Neil Hodgson
Uriel M.: Decorators should stay out of the language until there is a better syntax.
The current patch has been provided by Michael Sparks, and is available at http://www.python.org/sf/1013835. The previous minor scoping issue 25 was fixed on Aug 21, 2004.
pie syntax (CVS, Aug 17 2004):
decorator: '@' dotted_name [ '(' [arglist] ')' ] NEWLINE
decorators: decorator+
funcdef: [decorators] 'def' NAME parameters ':' suite
suite-and-keyword syntax:
decorator: dotted_name [ '(' [arglist] ')' ] NEWLINE
decorators: decorator+
funcdef: ['using' ':' (decorator |NEWLINE INDENT decorators DEDENT) ]
'def' NAME parameters ':'suite
"1) If we do decorators at all, decorators should be allowed to be arbitrary expressions.See also: http://mail.python.org/pipermail/python-dev/2004-April/043957.html
2) Since we allow arbitrary expressions, decorators provide a much better way to set function attributes than the current way."
"I've got a little bit of philosophy on the use of keywords vs. punctuation. While punctuation is concise, and often traditional for things like arithmetic operations, giving punctuation too much power can lead to loss of readability. (So yes, I regret significant trailing commas in print and tuples -- just a little bit.)"