The future of Python

I have received a number of questions and remarks about my keynote talk at EuroSciPy 2010, ranging from questions about technical details to an inquiry about the release date of Python 4.0! Rather than writing lengthy replies to everyone, I try to address all these issues here.

First of all, my intentions behind the keynote were

  1. Encourage scientists to look at new tools and developments that I believe to be important in the near future (Python 3, Cython) and at others that might become important to scientific applications (JIT compilers, alternative implementations).
  2. Make computational scientists think about future commodity hardware (which is what we use most of the time) and its implications for programming, in particular the generalization of massively parallel computing.
  3. Show that easy-to-use parallel programming paradigms, in particular deterministic ones, exist today. Computational scientists need to realize that MPI and OpenMP are not the last word on parallel programming.
  4. Make my ideas concrete by showing how they could be implemented in Python.

My “Python 4.0” is completely fictitious and will probably never exist in exactly that form. However, it is important to realize that it could be implemented right now. With the GIL-free Python implementations (Jython, IronPython), it would even be rather straightforward to implement. For CPython, any implementation not removing the GIL would probably be too inefficient to be of practical interest.

Most of the ingredients for implementing my “Python 4.0” are well-known and have already been used in other languages or libraries:

  • The “declarative concurrency” programming paradigm has been used in Oz and FlowJava, but to the best of my knowledge not in any mainstream programming language. It is explained very well in the book Concepts, Techniques, and Models of Computer Programming, by Peter van Roy and Seif Haridi, and also in the freely downloadable essay Programming paradigms for dummies. Basically, it is the functional paradigm extended with annotations that identify computations to be done in parallel. Remove those annotations, and you get plain functional programs that yield the same results. Declarative concurrency is free of deadlocks and race conditions, which I think is a critical property for any parallel programming paradigm to be considered for a high-level language such as Python. Another nice feature of declarative concurrency is that data parallelism, including nested data parallelism, is a special case that can be implemented on top of it. Data parallelism is a useful paradigm for many scientific applications.
  • Futures are asynchronous tasks provided as library functions for various languages. A Python library for futures is the subject of PEP 3148; an implementation is already available.
  • The importance of effect-free functions for all kinds of code transformations (automatic or not) is widely recognized. It is equally recognized that a useful program needs to have effects. The two basic approaches to dealing with this contradiction are (a) allow effects, but make it as easy as possible not to use them to encourage a mostly effect-free style and (b) design a language without effects (pure functional language) and provide a mechanism to put in effects with special annotation or syntax but clearly as an exceptional feature. The best-known language in the second category is Haskell with its use of monads for controlling effects. Most functional languages are in the first category.
  • Efficient data structures for functional programs have been a subject of research for quite a while and quite a few good ones are known. It would be straightforward to replace Python’s tuple implementation by something more efficient in typical functional settings, or to add an efficient immutable dictionary implementation. The standard reference is Chris Osaki’s book Purely Functional Data Structures.

Futures may seem to provide most of what declarative concurrency promises, but this is not quite true. Futures are objects representing computations. They have a method that client code must call to wait for the result and retrieve it. Since waiting is an explicit operation on a standard object, it is easy to create a situation in which two futures wait for each other: a deadlock. This can only be avoided by not having futures accessible as standard objects. The language implementation must recognize futures as special and insert a wait call before any access to the value of the result. For this reason, declarative concurrency cannot be implemented as a library.

Another important condition for implementing declarative concurrency with futures is that code inside a future must be effect-free. Otherwise multiple concurrently running futures can modify the same object and create a race condition.

Probably the only truly original contribution in my “Python 4.0” scenario is the dynamically verified effect-free subset of the Python language. Most languages, even functional ones, provide no way for a compiler or a run-time system to verify that a given function is effect-free. Haskell is perhaps the only exception in having a static type system that can identify effect-free code. In Python, that is not a viable approach because everything is dynamic. But why not provide at least a run-time check for effect-free code where useful? It’s still better to have a program crash with an exception saying “you did I/O in what should have been an effect-free function” than get wrong results silently.

Here is an outline of how such an approach could be implemented. Each function and method would have a flag saying “I am supposed to be effect-free.” In my examples, this flag is set by the decorator @noeffects, but other ways are possible. Built-in functions would of course have that flag set correctly as well. As soon as the interpreter enters a function marked as effect-free, it goes into “functional mode” until it returns from that function again. In functional mode, it raises an exception whenever an unflagged function or method is called.

Some details to consider:

  • Effect-free functions may not contain global or nonlocal statements. Probably the only way to enforce this is to have special syntax for defining effect-free functions and methods (rather than a decorator) and make those statements syntactically illegal inside.
  • It would be much more useful to have a “referentially transparent” subset rather than an “effect-free” subset, but this is also much harder to implement. A referentially transparent function guarantees to return the same result for the same input arguments, but may modify mutable objects that it has created internally. For example, a typical matrix inversion function allocates an array for its result and then uses an imperative algorithm that modifies the elements of that array repeatedly before returning it. Such a function can be used as an asynchronous task without risk, but its building blocks cannot be safely run concurrently.

Finally, a comment on a minor issue. I have been asked if the “async” keyword is strictly necessary. The answer is no, but it makes the code much more readable. The main role of async is to write a function call without having it executed immediately. The same problem occurs in callbacks in GUI programming: you have to specify a function call to be executed at a later time. The usual solution is a parameter-free lambda expression, and that same trick could be used to make async a function rather than a keyword. But readability suffers a lot.

Explore posts in the same categories: Programming

4 Comments on “The future of Python”

  1. Anonymous Says:

    What about CSP (communicating serial processes)? There’s a Python library for it: CSP doesn’t have a problem with the GIL.

    • khinsen Says:

      CSP is a non-deterministic programming paradigm. For that reason, it makes parallel programs much more difficult to write and understand, and introduces the possibility of hard-to-find bugs. There are applications where non-determinism is a requirement (e.g. servers that have to react to multiple incoming requests), but for most applications in science (and probably elsewhere) non-determinism is not necessary.

      Note also that the GIL is a separate issue. What the GIL does is prevent execution threads running in parallel to access shared data efficiently. As long as CPython has the GIL, it can reach parallelism only through multiple independent processes that communicate with each other. This entails a performance overhead, no matter which programming paradigm is used at the Python level. The “declarative concurrency” paradigm that I advocate can very well be implemented on top of CSP, or on top of the current multiprocessing module, but it will suffer from the same performance loss.

  2. Jordan Says:

    What is your opinion of building a concurrent.futures.Executor MPI implementation compliant with PEP3148?

    • khinsen Says:

      I am not sure this is a very useful approach, but I’d be happy to be proven wrong.

      MPI is made for distributed computing, meaning that both the data and the processes working on the data are distributed among several nodes with independent local memory. To be efficient, processes must work mostly on local data and only occasional exchange data with other nodes. Futures are packaged processes, originally designed to be used in a shared-memory environment (which is problematic in Python because of the GIL). Futures running in another process must carry all their data with them, i.e. the data is passed as arguments. That means a lot of communication, all the more if the data passes through a serialization bottleneck. All that means that futures are not a good way to use a machine set up for good MPI performance.

      Of course, if you have that machine, and have coarse-grained parallelism, then MPI-based futures are a way to use that machine easily. Perhaps there is a large enough application domain that makes it worth writing such an implementation. But I am not convinced.

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Twitter picture

You are commenting using your Twitter account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: