On the usage of Pythonic idioms (2018)

An enlightened Python master teaches disciples about the zen of Python

Design patterns are conventions that make it easier to communicate and reason about commonly occurring problems and their solutions. The Python community takes the idea of using conventions one step further, and argues that code should be “Pythonic”. Alexandru et al. explore what that means in practice.

Why it matters

The Python community cares so much about having a single “right” way to design and implement solutions, that it has even coined a special word for it: “Pythonic”.

The term appears to be very widespread: the word “Pythonic” is mentioned in a staggering 90% of all Stack Overflow threads about Python.

Nevertheless, the concept itself has never been thoroughly researched.

How the study was conducted

The authors aim to learn how well the concept of “Pythonic” code is understood by developers and how widespread concrete Pythonic idioms are.

First, they interviewed 13 Python developers with diverse backgrounds and levels of Python experience.

Then, the authors compiled a list of idioms by manually analysing presentations, books, and online resources by renowned Python developersMore info in the box below!.

Each of these idioms can be represented using an abstract syntax treeAn abstract syntax tree shows the “shape” of the solution without considering things like whitespace, newlines, and variable names.. The authors therefore mined 1,000 popular Python repositories on GitHub for these abstract syntax trees to determine how prevalent these idioms are in practice.

What discoveries were made

Correct usage of Pythonic idioms is considered to very important and sets expert Python developers apart from less experienced Python developers.

Developers

All interviewed developers were familiar with the term “Pythonic” and agreed that Pythonic code is desirable.

Pythonic code is generally understood to be elegant and readable code that makes good use of features offered by the language and its standard libraries. This often results in more succinct code, improved readability, and better performance.

Developers usually learn about the best way to write Python code by reading books and code written by others, attending conferences, discussions with colleagues, and consulting Stack Overflow.

Actually applying this knowledge to new code can be hard at first though, and may require years of practice before one can readily implement them. This is probably why the “Pythonic-ness” of code can be used to estimate its developer’s level of Python expertise.

Writing pythonic code is usually mandatory in open-source projects. However, code in closed-source projects often has lower standards and is more likely to be “merely” documentedIt’s not clear whether this means that developers working on closed-source projects are less skilled in Python or that it’s harder to accurately estimate skill levels for developers who work on closed-source projects..

Idioms

The table below shows the idioms that were found in the 1,000 projects, together with the number of projects in which they were used and the total number of usages across those projects.

Idiom # Projects Total
List comprehension 866 75,466
Dict comprehension 146 796
Generator expression 709 33,038
Decorator 765 114,545
Simple magic methods 759 78,376
Intermediate magic methods 417 13,255
Advanced magic methods 190 2,613
finally 504 18,900
with 848 143,453
enumerate 681 19,453
yield 672 56,687
lambda 664 109,600
collections.defaultdict 314 2,930
collections.namedtuple 262 2,211
collections.deque 180 1,698
collections.Counter 133 1,074
@classmethod 518 22,220
@staticmethod 487 11,552
zip 554 14,929
itertools 128 839
functools.total_ordering 30 82

You can find more thorough explanations of these idioms on the paper’s companion website.

Write better Python code

If you’re looking towards improving your Python skills you may want to take a look at some of the resources below.

Head First Python is probably the best resource for beginners, while Raymond Hettinger’s presentation is mostly useful as a refresher for developers who are already familiar with Python.

Online resources
Books

The important bits

  1. Pythonic code is elegant, readable, and makes good use of features offered by the language and its standard libraries
  2. Solutions that are implemented pythonically are often more readable and more performant
  3. A developer’s level of expertise can be estimated by their usage of Pythonic idioms
  4. Code in open-source projects is often more Pythonic than code in closed-source projects
  5. Check out the list of idioms in this article or on the paper’s companion website