PEP 554 – Multiple Interpreters in the Stdlib

PEP

554

Title

Multiple Interpreters in the Stdlib

Author

Eric Snow <ericsnowcurrently at gmail.com>

BDFL-Delegate

Antoine Pitrou <antoine at python.org>

Status

Draft

Type

Standards Track

Content-Type

text/x-rst

Created

2017-09-05

Python-Version

3.9

Post-History

07-Sep-2017, 08-Sep-2017, 13-Sep-2017, 05-Dec-2017, 09-May-2018

Contents

• Abstract

• A Disclaimer about the GIL

• Proposal

  • The “interpreters” Module

  • API summary for interpreters module

  • “Extending Python” Docs

• Examples

  • Run isolated code

  • Run in a thread

  • Pre-populate an interpreter

  • Handling an exception

  • Re-raising an exception

  • Synchronize using a channel

  • Sharing a file descriptor

  • Passing objects via marshal

  • Passing objects via pickle

  • Running a module

  • Running as script (including zip archives & directories)

  • Running in a thread pool executor

• Rationale

  • Concerns

• About Subinterpreters

  • Concurrency

  • Shared data

  • Interpreter Isolation

  • Existing Usage

• Provisional Status

• Alternate Python Implementations

• “interpreters” Module API

  • Uncaught Exceptions

  • API for sharing data

  • Channel Association

• Open Questions

• Deferred Functionality

  • Interpreter.call()

  • timeout arg to recv() and send()

  • get_main()

  • Interpreter.run_in_thread()

  • Synchronization Primitives

  • CSP Library

  • Syntactic Support

  • Multiprocessing

  • C-extension opt-in/opt-out

  • Poisoning channels

  • Resetting __main__

  • Resetting an interpreter’s state

  • File descriptors and sockets in channels

  • Integration with async

  • Support for iteration

  • Channel context managers

  • Pipes and Queues

  • Buffering

  • Return a lock from send()

  • Add a “reraise” method to RunFailedError

  • Support prioritization in channels

  • Support inheriting settings (and more?)

• Rejected Ideas

  • Explicit channel association

  • Use pipes instead of channels

  • Use queues instead of channels

  • “enumerate”

  • Alternate solutions to prevent leaking exceptions across interpreters

  • Always associate each new interpreter with its own thread

  • Only associate interpreters upon use

• Implementation

• References

• Copyright

Abstract

CPython has supported multiple interpreters in the same process (AKA “subinterpreters”) since version 1.5 (1997). The feature has been available via the C-API. [c-api] Subinterpreters operate in relative isolation from one another, which facilitates novel alternative approaches to concurrency.

This proposal introduces the stdlib interpreters module. The module will be provisional. It exposes the basic functionality of subinterpreters already provided by the C-API, along with new (basic) functionality for sharing data between interpreters.

A Disclaimer about the GIL

To avoid any confusion up front: This PEP is unrelated to any efforts to stop sharing the GIL between subinterpreters. At most this proposal will allow users to take advantage of any results of work on the GIL. The position here is that exposing subinterpreters to Python code is worth doing, even if they still share the GIL.

Proposal

The interpreters module will be added to the stdlib. To help authors of extension modules, a new page will be added to the Extending Python docs. More information on both is found in the immediately following sections.

The “interpreters” Module

The interpreters module will provide a high-level interface to subinterpreters and wrap a new low-level _interpreters (in the same way as the threading module). See the Examples section for concrete usage and use cases.

Along with exposing the existing (in CPython) subinterpreter support, the module will also provide a mechanism for sharing data between interpreters. This mechanism centers around “channels”, which are similar to queues and pipes.

Note that objects are not shared between interpreters since they are tied to the interpreter in which they were created. Instead, the objects’ data is passed between interpreters. See the Shared data section for more details about sharing between interpreters.

At first only the following types will be supported for sharing:

• None

• bytes

• str

• int

• PEP 3118 buffer objects (via send_buffer())

• PEP 554 channels

Support for other basic types (e.g. bool, float, Ellipsis) will be added later.

API summary for interpreters module

Here is a summary of the API for the interpreters module. For a more in-depth explanation of the proposed classes and functions, see the “interpreters” Module API section below.

For creating and using interpreters:

signature	description
list_all() -> [Interpreter]	Get all existing interpreters.
get_current() -> Interpreter	Get the currently running interpreter.
get_main() -> Interpreter	Get the main interpreter.
create() -> Interpreter	Initialize a new (idle) Python interpreter.

signature	description
class Interpreter(id)	A single interpreter.
.id	The interpreter’s ID (read-only).
.is_running() -> bool	Is the interpreter currently executing code?
.destroy()	Finalize and destroy the interpreter.
.run(src_str, /, *, channels=None)	Run the given source code in the interpreter. (This blocks the current thread until done.)

exception	base	description
RunFailedError	RuntimeError	Interpreter.run() resulted in an uncaught exception.

For sharing data between interpreters:

signature	description
is_shareable(obj) -> Bool	Can the object’s data be shared between interpreters?
create_channel() -> (RecvChannel, SendChannel)	Create a new channel for passing data between interpreters.
list_all_channels() -> [(RecvChannel, SendChannel)]	Get all open channels.

signature	description
class RecvChannel(id)	The receiving end of a channel.
.id	The channel’s unique ID.
.interpreters	The list of associated interpreters.
.recv() -> object	Get the next object from the channel, and wait if none have been sent. Associate the interpreter with the channel.
.recv_nowait(default=None) -> object	Like recv(), but return the default instead of waiting.
.release()	No longer associate the current interpreter with the channel (on the receiving end).
.close(force=False)	Close the channel in all interpreters.

signature	description
class SendChannel(id)	The sending end of a channel.
.id	The channel’s unique ID.
.interpreters	The list of associated interpreters.
.send(obj)	Send the object (i.e. its data) to the receiving end of the channel and wait. Associate the interpreter with the channel.
.send_nowait(obj)	Like send(), but return False if not received.
.send_buffer(obj)	Send the object’s (PEP 3118) buffer to the receiving end of the channel and wait. Associate the interpreter with the channel.
.send_buffer_nowait(obj)	Like send_buffer(), but return False if not received.
.release()	No longer associate the current interpreter with the channel (on the sending end).
.close(force=False)	Close the channel in all interpreters.

exception	base	description
ChannelError	Exception	The base class for channel-related exceptions.
ChannelNotFoundError	ChannelError	The identified channel was not found.
ChannelEmptyError	ChannelError	The channel was unexpectedly empty.
ChannelNotEmptyError	ChannelError	The channel was unexpectedly not empty.
NotReceivedError	ChannelError	Nothing was waiting to receive a sent object.
ChannelClosedError	ChannelError	The channel is closed.
ChannelReleasedError	ChannelClosedError	The channel is released (but not yet closed).

“Extending Python” Docs

Many extension modules do not support use in subinterpreters. The authors and users of such extension modules will both benefit when they are updated to support subinterpreters. To help with that, a new page will be added to the Extending Python docs.

This page will explain how to implement PEP 489 support and how to move from global module state to per-interpreter.

Examples

Run isolated code

interp = interpreters.create()
print('before')
interp.run('print("during")')
print('after')

Run in a thread

interp = interpreters.create()
def run():
    interp.run('print("during")')
t = threading.Thread(target=run)
print('before')
t.start()
print('after')

Pre-populate an interpreter

interp = interpreters.create()
interp.run(tw.dedent("""
    import some_lib
    import an_expensive_module
    some_lib.set_up()
    """))
wait_for_request()
interp.run(tw.dedent("""
    some_lib.handle_request()
    """))

Handling an exception

interp = interpreters.create()
try:
    interp.run(tw.dedent("""
        raise KeyError
        """))
except interpreters.RunFailedError as exc:
    print(f"got the error from the subinterpreter: {exc}")

Re-raising an exception

interp = interpreters.create()
try:
    try:
        interp.run(tw.dedent("""
            raise KeyError
            """))
    except interpreters.RunFailedError as exc:
        raise exc.__cause__
except KeyError:
    print("got a KeyError from the subinterpreter")

Note that this pattern is a candidate for later improvement.

Synchronize using a channel

interp = interpreters.create()
r, s = interpreters.create_channel()
def run():
    interp.run(tw.dedent("""
        reader.recv()
        print("during")
        reader.release()
        """),
        shared=dict(
            reader=r,
            ),
        )
t = threading.Thread(target=run)
print('before')
t.start()
print('after')
s.send(b'')
s.release()

Sharing a file descriptor

interp = interpreters.create()
r1, s1 = interpreters.create_channel()
r2, s2 = interpreters.create_channel()
def run():
    interp.run(tw.dedent("""
        fd = int.from_bytes(
                reader.recv(), 'big')
        for line in os.fdopen(fd):
            print(line)
        writer.send(b'')
        """),
        shared=dict(
            reader=r,
            writer=s2,
            ),
        )
t = threading.Thread(target=run)
t.start()
with open('spamspamspam') as infile:
    fd = infile.fileno().to_bytes(1, 'big')
    s.send(fd)
    r.recv()

Passing objects via marshal

interp = interpreters.create()
r, s = interpreters.create_channel()
interp.run(tw.dedent("""
    import marshal
    """),
    shared=dict(
        reader=r,
        ),
    )
def run():
    interp.run(tw.dedent("""
        data = reader.recv()
        while data:
            obj = marshal.loads(data)
            do_something(obj)
            data = reader.recv()
        reader.release()
        """))
t = threading.Thread(target=run)
t.start()
for obj in input:
    data = marshal.dumps(obj)
    s.send(data)
s.send(None)

Passing objects via pickle

interp = interpreters.create()
r, s = interpreters.create_channel()
interp.run(tw.dedent("""
    import pickle
    """),
    shared=dict(
        reader=r,
        ),
    )
def run():
    interp.run(tw.dedent("""
        data = reader.recv()
        while data:
            obj = pickle.loads(data)
            do_something(obj)
            data = reader.recv()
        reader.release()
        """))
t = threading.Thread(target=run)
t.start()
for obj in input:
    data = pickle.dumps(obj)
    s.send(data)
s.send(None)

Running a module

interp = interpreters.create()
main_module = mod_name
interp.run(f'import runpy; runpy.run_module({main_module!r})')

Running as script (including zip archives & directories)

interp = interpreters.create()
main_script = path_name
interp.run(f"import runpy; runpy.run_path({main_script!r})")

Running in a thread pool executor

interps = [interpreters.create() for i in range(5)]
with concurrent.futures.ThreadPoolExecutor(max_workers=len(interps)) as pool:
    print('before')
    for interp in interps:
        pool.submit(interp.run, 'print("starting"); print("stopping")'
    print('after')

Rationale

Running code in multiple interpreters provides a useful level of isolation within the same process. This can be leveraged in a number of ways. Furthermore, subinterpreters provide a well-defined framework in which such isolation may extended.

Nick Coghlan explained some of the benefits through a comparison with multi-processing [benefits]:

[I] expect that communicating between subinterpreters is going
to end up looking an awful lot like communicating between
subprocesses via shared memory.

The trade-off between the two models will then be that one still
just looks like a single process from the point of view of the
outside world, and hence doesn't place any extra demands on the
underlying OS beyond those required to run CPython with a single
interpreter, while the other gives much stricter isolation
(including isolating C globals in extension modules), but also
demands much more from the OS when it comes to its IPC
capabilities.

The security risk profiles of the two approaches will also be quite
different, since using subinterpreters won't require deliberately
poking holes in the process isolation that operating systems give
you by default.

CPython has supported subinterpreters, with increasing levels of support, since version 1.5. While the feature has the potential to be a powerful tool, subinterpreters have suffered from neglect because they are not available directly from Python. Exposing the existing functionality in the stdlib will help reverse the situation.

This proposal is focused on enabling the fundamental capability of multiple isolated interpreters in the same Python process. This is a new area for Python so there is relative uncertainly about the best tools to provide as companions to subinterpreters. Thus we minimize the functionality we add in the proposal as much as possible.

Concerns

• “subinterpreters are not worth the trouble”

Some have argued that subinterpreters do not add sufficient benefit to justify making them an official part of Python. Adding features to the language (or stdlib) has a cost in increasing the size of the language. So an addition must pay for itself. In this case, subinterpreters provide a novel concurrency model focused on isolated threads of execution. Furthermore, they provide an opportunity for changes in CPython that will allow simultaneous use of multiple CPU cores (currently prevented by the GIL).

Alternatives to subinterpreters include threading, async, and multiprocessing. Threading is limited by the GIL and async isn’t the right solution for every problem (nor for every person). Multiprocessing is likewise valuable in some but not all situations. Direct IPC (rather than via the multiprocessing module) provides similar benefits but with the same caveat.

Notably, subinterpreters are not intended as a replacement for any of the above. Certainly they overlap in some areas, but the benefits of subinterpreters include isolation and (potentially) performance. In particular, subinterpreters provide a direct route to an alternate concurrency model (e.g. CSP) which has found success elsewhere and will appeal to some Python users. That is the core value that the interpreters module will provide.

• “stdlib support for subinterpreters adds extra burden on C extension authors”

In the Interpreter Isolation section below we identify ways in which isolation in CPython’s subinterpreters is incomplete. Most notable is extension modules that use C globals to store internal state. PEP 3121 and PEP 489 provide a solution for most of the problem, but one still remains. [petr-c-ext] Until that is resolved (see PEP 573), C extension authors will face extra difficulty to support subinterpreters.

Consequently, projects that publish extension modules may face an increased maintenance burden as their users start using subinterpreters, where their modules may break. This situation is limited to modules that use C globals (or use libraries that use C globals) to store internal state. For numpy, the reported-bug rate is one every 6 months. [bug-rate]

Ultimately this comes down to a question of how often it will be a problem in practice: how many projects would be affected, how often their users will be affected, what the additional maintenance burden will be for projects, and what the overall benefit of subinterpreters is to offset those costs. The position of this PEP is that the actual extra maintenance burden will be small and well below the threshold at which subinterpreters are worth it.

• “creating a new concurrency API deserves much more thought and experimentation, so the new module shouldn’t go into the stdlib right away, if ever”

Introducing an API for a a new concurrency model, like happened with asyncio, is an extremely large project that requires a lot of careful consideration. It is not something that can be done a simply as this PEP proposes and likely deserves significant time on PyPI to mature. (See Nathaniel’s post on python-dev.)

However, this PEP does not propose any new concurrency API. At most it exposes minimal tools (e.g. subinterpreters, channels) which may be used to write code that follows patterns associated with (relatively) new-to-Python concurrency models. Those tools could also be used as the basis for APIs for such concurrency models. Again, this PEP does not propose any such API.

• “there is no point to exposing subinterpreters if they still share the GIL”

• “the effort to make the GIL per-interpreter is disruptive and risky”

A common misconception is that this PEP also includes a promise that subinterpreters will no longer share the GIL. When that is clarified, the next question is “what is the point?”. This is already answered at length in this PEP. Just to be clear, the value lies in:

* increase exposure of the existing feature, which helps improve
  the code health of the entire CPython runtime
* expose the (mostly) isolated execution of subinterpreters
* preparation for per-interpreter GIL
* encourage experimentation

About Subinterpreters

Concurrency

Concurrency is a challenging area of software development. Decades of research and practice have led to a wide variety of concurrency models, each with different goals. Most center on correctness and usability.

One class of concurrency models focuses on isolated threads of execution that interoperate through some message passing scheme. A notable example is Communicating Sequential Processes (CSP) (upon which Go’s concurrency is roughly based). The isolation inherent to subinterpreters makes them well-suited to this approach.

Shared data

Subinterpreters are inherently isolated (with caveats explained below), in contrast to threads. So the same communicate-via-shared-memory approach doesn’t work. Without an alternative, effective use of concurrency via subinterpreters is significantly limited.

The key challenge here is that sharing objects between interpreters faces complexity due to various constraints on object ownership, visibility, and mutability. At a conceptual level it’s easier to reason about concurrency when objects only exist in one interpreter at a time. At a technical level, CPython’s current memory model limits how Python objects may be shared safely between interpreters; effectively objects are bound to the interpreter in which they were created. Furthermore the complexity of object sharing increases as subinterpreters become more isolated, e.g. after GIL removal.

Consequently,the mechanism for sharing needs to be carefully considered. There are a number of valid solutions, several of which may be appropriate to support in Python. This proposal provides a single basic solution: “channels”. Ultimately, any other solution will look similar to the proposed one, which will set the precedent. Note that the implementation of Interpreter.run() will be done in a way that allows for multiple solutions to coexist, but doing so is not technically a part of the proposal here.

Regarding the proposed solution, “channels”, it is a basic, opt-in data sharing mechanism that draws inspiration from pipes, queues, and CSP’s channels. [fifo]

As simply described earlier by the API summary, channels have two operations: send and receive. A key characteristic of those operations is that channels transmit data derived from Python objects rather than the objects themselves. When objects are sent, their data is extracted. When the “object” is received in the other interpreter, the data is converted back into an object owned by that interpreter.

To make this work, the mutable shared state will be managed by the Python runtime, not by any of the interpreters. Initially we will support only one type of objects for shared state: the channels provided by create_channel(). Channels, in turn, will carefully manage passing objects between interpreters.

This approach, including keeping the API minimal, helps us avoid further exposing any underlying complexity to Python users. Along those same lines, we will initially restrict the types that may be passed through channels to the following:

• None

• bytes

• str

• int

• PEP 3118 buffer objects (via send_buffer())

• channels

Limiting the initial shareable types is a practical matter, reducing the potential complexity of the initial implementation. There are a number of strategies we may pursue in the future to expand supported objects and object sharing strategies.

Interpreter Isolation

CPython’s interpreters are intended to be strictly isolated from each other. Each interpreter has its own copy of all modules, classes, functions, and variables. The same applies to state in C, including in extension modules. The CPython C-API docs explain more. [caveats]

However, there are ways in which interpreters share some state. First of all, some process-global state remains shared:

• file descriptors

• builtin types (e.g. dict, bytes)

• singletons (e.g. None)

• underlying static module data (e.g. functions) for builtin/extension/frozen modules

There are no plans to change this.

Second, some isolation is faulty due to bugs or implementations that did not take subinterpreters into account. This includes things like extension modules that rely on C globals. [cryptography] In these cases bugs should be opened (some are already):

• readline module hook functions (http://bugs.python.org/issue4202)

• memory leaks on re-init (http://bugs.python.org/issue21387)

Finally, some potential isolation is missing due to the current design of CPython. Improvements are currently going on to address gaps in this area:

• GC is not run per-interpreter [global-gc]

• at-exit handlers are not run per-interpreter [global-atexit]

• extensions using the PyGILState_* API are incompatible [gilstate]

• interpreters share memory management (e.g. allocators, gc)

• interpreters share the GIL

Existing Usage

Subinterpreters are not a widely used feature. In fact, the only documented cases of wide-spread usage are mod_wsgi, OpenStack Ceph, and JEP. On the one hand, these cases provide confidence that existing subinterpreter support is relatively stable. On the other hand, there isn’t much of a sample size from which to judge the utility of the feature.

Provisional Status

The new interpreters module will be added with “provisional” status (see PEP 411). This allows Python users to experiment with the feature and provide feedback while still allowing us to adjust to that feedback. The module will be provisional in Python 3.8 and we will make a decision before the 3.9 release whether to keep it provisional, graduate it, or remove it.

Alternate Python Implementations

I’ve solicited feedback from various Python implementors about support for subinterpreters. Each has indicated that they would be able to support subinterpreters (if they choose to) without a lot of trouble. Here are the projects I contacted:

• jython ([jython])

• ironpython (personal correspondence)

• pypy (personal correspondence)

• micropython (personal correspondence)

“interpreters” Module API

The module provides the following functions:

list_all() -> [Interpreter]

   Return a list of all existing interpreters.

get_current() => Interpreter

   Return the currently running interpreter.

get_main() => Interpreter

   Return the main interpreter.

create() -> Interpreter

   Initialize a new Python interpreter and return it.  The
   interpreter will be created in the current thread and will remain
   idle until something is run in it.  The interpreter may be used
   in any thread and will run in whichever thread calls
   ``interp.run()``.

The module also provides the following class:

class Interpreter(id):

   id -> int:

      The interpreter's ID (read-only).

   is_running() -> bool:

      Return whether or not the interpreter is currently executing
      code.  Calling this on the current interpreter will always
      return True.

   destroy():

      Finalize and destroy the interpreter.

      This may not be called on an already running interpreter.
      Doing so results in a RuntimeError.

   run(source_str, /, *, channels=None):

      Run the provided Python source code in the interpreter.  If
      the "channels" keyword argument is provided (and is a mapping
      of attribute names to channels) then it is added to the
      interpreter's execution namespace (the interpreter's
      "__main__" module).  If any of the values are not RecvChannel
      or SendChannel instances then ValueError gets raised.

      This may not be called on an already running interpreter.
      Doing so results in a RuntimeError.

      A "run()" call is similar to a function call.  Once it
      completes, the code that called "run()" continues executing
      (in the original interpreter).  Likewise, if there is any
      uncaught exception then it effectively (see below) propagates
      into the code where ``run()`` was called.  However, unlike
      function calls (but like threads), there is no return value.
      If any value is needed, pass it out via a channel.

      The big difference from functions is that "run()" executes
      the code in an entirely different interpreter, with entirely
      separate state.  The state of the current interpreter in the
      current OS thread is swapped out with the state of the target
      interpreter (the one that will execute the code).  When the
      target finishes executing, the original interpreter gets
      swapped back in and its execution resumes.

      So calling "run()" will effectively cause the current Python
      thread to pause.  Sometimes you won't want that pause, in
      which case you should make the "run()" call in another thread.
      To do so, add a function that calls "run()" and then run that
      function in a normal "threading.Thread".

      Note that the interpreter's state is never reset, neither
      before "run()" executes the code nor after.  Thus the
      interpreter state is preserved between calls to "run()".
      This includes "sys.modules", the "builtins" module, and the
      internal state of C extension modules.

      Also note that "run()" executes in the namespace of the
      "__main__" module, just like scripts, the REPL, "-m", and
      "-c".  Just as the interpreter's state is not ever reset, the
      "__main__" module is never reset.  You can imagine
      concatenating the code from each "run()" call into one long
      script.  This is the same as how the REPL operates.

      Supported code: source text.

Uncaught Exceptions

Regarding uncaught exceptions in Interpreter.run(), we noted that they are “effectively” propagated into the code where run() was called. To prevent leaking exceptions (and tracebacks) between interpreters, we create a surrogate of the exception and its traceback (see traceback.TracebackException), set it to __cause__ on a new RunFailedError, and raise that.

Raising (a proxy of) the exception directly is problematic since it’s harder to distinguish between an error in the run() call and an uncaught exception from the subinterpreter.

API for sharing data

Subinterpreters are less useful without a mechanism for sharing data between them. Sharing actual Python objects between interpreters, however, has enough potential problems that we are avoiding support for that here. Instead, only mimimum set of types will be supported. Initially this will include None, bytes, str, int, and channels. Further types may be supported later.

The interpreters module provides a function that users may call to determine whether an object is shareable or not:

is_shareable(obj) -> bool:

   Return True if the object may be shared between interpreters.
   This does not necessarily mean that the actual objects will be
   shared.  Insead, it means that the objects' underlying data will
   be shared in a cross-interpreter way, whether via a proxy, a
   copy, or some other means.

This proposal provides two ways to share such objects between interpreters.

First, channels may be passed to run() via the channels keyword argument, where they are effectively injected into the target interpreter’s __main__ module. While passing arbitrary shareable objects this way is possible, doing so is mainly intended for sharing meta-objects (e.g. channels) between interpreters. It is less useful to pass other objects (like bytes) to run directly.

Second, the main mechanism for sharing objects (i.e. their data) between interpreters is through channels. A channel is a simplex FIFO similar to a pipe. The main difference is that channels can be associated with zero or more interpreters on either end. Like queues, which are also many-to-many, channels are buffered (though they also offer methods with unbuffered semantics).

Python objects are not shared between interpreters. However, in some cases data those objects wrap is actually shared and not just copied. One example is PEP 3118 buffers. In those cases the object in the original interpreter is kept alive until the shared data in the other interpreter is no longer used. Then object destruction can happen like normal in the original interpreter, along with the previously shared data.

The interpreters module provides the following functions related to channels:

create_channel() -> (RecvChannel, SendChannel):

   Create a new channel and return (recv, send), the RecvChannel
   and SendChannel corresponding to the ends of the channel.  The
   lifetime of the channel is determined by associations between
   intepreters and the channel's ends (see below).

   Both ends of the channel are supported "shared" objects (i.e.
   may be safely shared by different interpreters.  Thus they
   may be passed as keyword arguments to "Interpreter.run()".

list_all_channels() -> [(RecvChannel, SendChannel)]:

   Return a list of all open channel-end pairs.

The module also provides the following channel-related classes:

class RecvChannel(id):

   The receiving end of a channel.  An interpreter may use this to
   receive objects from another interpreter.  At first only a few
   of the simple, immutable builtin types will be supported.

   id -> int:

      The channel's unique ID.  This is shared with the "send" end.

   interpreters => [Interpreter]:

      The list of interpreters associated with the "recv" end of
      the channel.  (See below for more on how interpreters are
      associated with channels.)  If the channel has been closed
      then raise ChannelClosedError.

   recv():

      Return the next object (i.e. the data from the sent object)
      from the channel.  If none have been sent then wait until
      the next send.

      If the channel is already closed then raise ChannelClosedError.
      If the channel isn't closed but the current interpreter already
      called the "release()" method for the "recv" end then raise
      ChannelReleasedError (which is a subclass of
      ChannelClosedError).

   recv_nowait(default=None):

      Return the next object from the channel.  If none have been
      sent then return the default.  Otherwise, this is the same
      as the "recv()" method.

   release() -> bool:

      No longer associate the current interpreter with the channel
      (on the "recv" end) and block any future association If the
      interpreter was never associated with the channel then still
      block any future association.  The "send" end of the channel
      is unaffected by a released "recv" end.

      Once an interpreter is no longer associated with the "recv"
      end of the channel, any "recv()" and "recv_nowait()" calls
      from that interpreter will fail (even ongoing calls).  See
      "recv()" for details.

      See below for more on how association relates to auto-closing
      a channel.

      This operation is idempotent.  Return True if "release()"
      has not been called before by the current interpreter.

   close(force=False):

      Close both ends of the channel (in all interpreters).  This
      means that any further use of the channel anywhere raises
      ChannelClosedError.  If the channel is not empty then
      raise ChannelNotEmptyError (if "force" is False) or
      discard the remaining objects (if "force" is True)
      and close it.  Note that the behavior of closing
      the "send" end is slightly different.


class SendChannel(id):

   The sending end of a channel.  An interpreter may use this to
   send objects to another interpreter.  At first only a few of
   the simple, immutable builtin types will be supported.

   id -> int:

      The channel's unique ID.  This is shared with the "recv" end.

   interpreters -> [Interpreter]:

      Like "RecvChannel.interpreters" but for the "send" end.

   send(obj):

      Send the object (i.e. its data) to the "recv" end of the
      channel.  Wait until the object is received.  If the object
      is not shareable then ValueError is raised.

      If this channel end was already released
      by the interpreter then raise ChannelReleasedError.  If
      the channel is already closed then raise
      ChannelClosedError.

   send_nowait(obj):

      Send the object to the "recv" end of the channel.  This
      behaves the same as "send()", except for the waiting part.
      If no interpreter is currently receiving (waiting on the
      other end) then queue the object and return False.  Otherwise
      return True.

   send_buffer(obj):

      Send a MemoryView of the object rather than the object.
      Otherwise this is the same as "send()".  Note that the
      object must implement the PEP 3118 buffer protocol.
      The buffer will always be released in the original
      interpreter, like normal.

   send_buffer_nowait(obj):

      Send a MemoryView of the object rather than the object.
      If the other end is not currently receiving then return
      False.  Otherwise return True.

   release():

      This is the same as "RecvChannel.release(), but applied
      to the sending end of the channel.

   close(force=False):

      Close both ends of the channel (in all interpreters).  No
      matter what the "send" end of the channel is immediately
      closed.  If the channel is empty then close the "recv"
      end immediately too.  Otherwise, if "force" if False,
      close the "recv" end (and hence the full channel)
      once the channel becomes empty; or, if "force"
      is True, discard the remaining items and
      close immediately.

Note that send_buffer() is similar to how multiprocessing.Connection works. [mp-conn]

Channel Association

Each end (send/recv) of each channel is associated with a set of interpreters. This association effectively means “the channel end is available to that interpreter”. It has ramifications on introspection and on how channels are automatically closed.

When a channel is created, both ends are immediately associated with the current interpreter. When a channel end is passed to an interpreter via Interpreter.run(..., channels=...) then that interpreter is associated with the channel end. Likewise when a channel end is sent through another channel, the receiving interpreter is associated with the sent channel end.

A channel end is explicitly released by an interpreter through the release() method. It is also done automatically for an interpreter when the last *Channel object for the end in that interpreter is garbage-collected, as though release() were called.

Calling *Channel.close() automatically releases the channel in all interpreters for both ends.

Once the number of associated interpreters on both ends drops to 0, the channel is actually closed. The Python runtime will garbage-collect all closed channels, though it may not happen immediately.

Consequently, *Channel.interpreters means those to which the channel end was sent, still hold a reference to the channel end, and haven’t called release().

Open Questions

• add a “tp_share” type slot instead of using a global registry for shareable types?

• impact of data sharing on cache performance in multi-core scenarios? (see [cache-line-ping-pong])

• strictly disallow subinterpreter import of extension modules without PEP 489 support?

• add “isolated” mode to subinterpreters API?

An “isolated” mode for subinterpreters would mean an interpreter in that mode is especially restricted. It might include any of the following:

* ImportError when importing ext. module without PEP 489 support
* no daemon threads
* no threads at all
* no multiprocessing

For now the default would be False, but it would become True later.

• add a shareable synchronization primitive?

This would be _threading.Lock (or something like it) where interpreters would actually share the underlying mutex. This would provide much better efficiency than blocking channel ops. The main concern is that locks and channels don’t mix well (as learned in Go).

• add readiness callback support to channels?

This is an alternative to channel buffering. It is probably unnecessary, but may have enough advantages to consider it for the high-level API. It may also be better only for the low-level implementation.

• also track which interpreters are using a channel end?

Deferred Functionality

In the interest of keeping this proposal minimal, the following functionality has been left out for future consideration. Note that this is not a judgement against any of said capability, but rather a deferment. That said, each is arguably valid.

Interpreter.call()

It would be convenient to run existing functions in subinterpreters directly. Interpreter.run() could be adjusted to support this or a call() method could be added:

Interpreter.call(f, *args, **kwargs)

This suffers from the same problem as sharing objects between interpreters via queues. The minimal solution (running a source string) is sufficient for us to get the feature out where it can be explored.

timeout arg to recv() and send()

Typically functions that have a block argument also have a timeout argument. It sometimes makes sense to do likewise for functions that otherwise block, like the channel recv() and send() methods. We can add it later if needed.

get_main()

CPython has a concept of a “main” interpreter. This is the initial interpreter created during CPython’s runtime initialization. It may be useful to identify the main interpreter. For instance, the main interpreter should not be destroyed. However, for the basic functionality of a high-level API a get_main() function is not necessary. Furthermore, there is no requirement that a Python implementation have a concept of a main interpreter. So until there’s a clear need we’ll leave get_main() out.

Interpreter.run_in_thread()

This method would make a run() call for you in a thread. Doing this using only threading.Thread and run() is relatively trivial so we’ve left it out.

Synchronization Primitives

The threading module provides a number of synchronization primitives for coordinating concurrent operations. This is especially necessary due to the shared-state nature of threading. In contrast, subinterpreters do not share state. Data sharing is restricted to channels, which do away with the need for explicit synchronization. If any sort of opt-in shared state support is added to subinterpreters in the future, that same effort can introduce synchronization primitives to meet that need.

CSP Library

A csp module would not be a large step away from the functionality provided by this PEP. However, adding such a module is outside the minimalist goals of this proposal.

Syntactic Support

The Go language provides a concurrency model based on CSP, so it’s similar to the concurrency model that subinterpreters support. However, Go also provides syntactic support, as well several builtin concurrency primitives, to make concurrency a first-class feature. Conceivably, similar syntactic (and builtin) support could be added to Python using subinterpreters. However, that is way outside the scope of this PEP!

Multiprocessing

The multiprocessing module could support subinterpreters in the same way it supports threads and processes. In fact, the module’s maintainer, Davin Potts, has indicated this is a reasonable feature request. However, it is outside the narrow scope of this PEP.

C-extension opt-in/opt-out

By using the PyModuleDef_Slot introduced by PEP 489, we could easily add a mechanism by which C-extension modules could opt out of support for subinterpreters. Then the import machinery, when operating in a subinterpreter, would need to check the module for support. It would raise an ImportError if unsupported.

Alternately we could support opting in to subinterpreter support. However, that would probably exclude many more modules (unnecessarily) than the opt-out approach. Also, note that PEP 489 defined that an extension’s use of the PEP’s machinery implies support for subinterpreters.

The scope of adding the ModuleDef slot and fixing up the import machinery is non-trivial, but could be worth it. It all depends on how many extension modules break under subinterpreters. Given that there are relatively few cases we know of through mod_wsgi, we can leave this for later.

Poisoning channels

CSP has the concept of poisoning a channel. Once a channel has been poisoned, any send() or recv() call on it would raise a special exception, effectively ending execution in the interpreter that tried to use the poisoned channel.

This could be accomplished by adding a poison() method to both ends of the channel. The close() method can be used in this way (mostly), but these semantics are relatively specialized and can wait.

Resetting __main__

As proposed, every call to Interpreter.run() will execute in the namespace of the interpreter’s existing __main__ module. This means that data persists there between run() calls. Sometimes this isn’t desirable and you want to execute in a fresh __main__. Also, you don’t necessarily want to leak objects there that you aren’t using any more.

Note that the following won’t work right because it will clear too much (e.g. __name__ and the other “__dunder__” attributes:

interp.run('globals().clear()')

Possible solutions include:

• a create() arg to indicate resetting __main__ after each run call

• an Interpreter.reset_main flag to support opting in or out after the fact

• an Interpreter.reset_main() method to opt in when desired

• importlib.util.reset_globals() [reset_globals]

Also note that resetting __main__ does nothing about state stored in other modules. So any solution would have to be clear about the scope of what is being reset. Conceivably we could invent a mechanism by which any (or every) module could be reset, unlike reload() which does not clear the module before loading into it. Regardless, since __main__ is the execution namespace of the interpreter, resetting it has a much more direct correlation to interpreters and their dynamic state than does resetting other modules. So a more generic module reset mechanism may prove unnecessary.

This isn’t a critical feature initially. It can wait until later if desirable.

Resetting an interpreter’s state

It may be nice to re-use an existing subinterpreter instead of spinning up a new one. Since an interpreter has substantially more state than just the __main__ module, it isn’t so easy to put an interpreter back into a pristine/fresh state. In fact, there may be parts of the state that cannot be reset from Python code.

A possible solution is to add an Interpreter.reset() method. This would put the interpreter back into the state it was in when newly created. If called on a running interpreter it would fail (hence the main interpreter could never be reset). This would likely be more efficient than creating a new subinterpreter, though that depends on what optimizations will be made later to subinterpreter creation.

While this would potentially provide functionality that is not otherwise available from Python code, it isn’t a fundamental functionality. So in the spirit of minimalism here, this can wait. Regardless, I doubt it would be controversial to add it post-PEP.

File descriptors and sockets in channels

Given that file descriptors and sockets are process-global resources, support for passing them through channels is a reasonable idea. They would be a good candidate for the first effort at expanding the types that channels support. They aren’t strictly necessary for the initial API.

Integration with async

Per Antoine Pitrou [async]:

Has any thought been given to how FIFOs could integrate with async
code driven by an event loop (e.g. asyncio)?  I think the model of
executing several asyncio (or Tornado) applications each in their
own subinterpreter may prove quite interesting to reconcile multi-
core concurrency with ease of programming.  That would require the
FIFOs to be able to synchronize on something an event loop can wait
on (probably a file descriptor?).

A possible solution is to provide async implementations of the blocking channel methods (recv(), and send()). However, the basic functionality of subinterpreters does not depend on async and can be added later.

Support for iteration

Supporting iteration on RecvChannel (via __iter__() or _next__()) may be useful. A trivial implementation would use the recv() method, similar to how files do iteration. Since this isn’t a fundamental capability and has a simple analog, adding iteration support can wait until later.

Channel context managers

Context manager support on RecvChannel and SendChannel may be helpful. The implementation would be simple, wrapping a call to close() (or maybe release()) like files do. As with iteration, this can wait.

Pipes and Queues

With the proposed object passing machanism of “channels”, other similar basic types aren’t required to achieve the minimal useful functionality of subinterpreters. Such types include pipes (like channels, but one-to-one) and queues (like channels, but more generic). See below in Rejected Ideas for more information.

Even though these types aren’t part of this proposal, they may still be useful in the context of concurrency. Adding them later is entirely reasonable. The could be trivially implemented as wrappers around channels. Alternatively they could be implemented for efficiency at the same low level as channels.

Buffering

The proposed channels are unbuffered. This simplifies the API and implementation. If buffering is desirable we can add it later.

Return a lock from send()

When sending an object through a channel, you don’t have a way of knowing when the object gets received on the other end. One way to work around this is to return a locked threading.Lock from SendChannel.send() that unlocks once the object is received.

Alternately, the proposed SendChannel.send() (blocking) and SendChannel.send_nowait() provide an explicit distinction that is less likely to confuse users.

Note that returning a lock would matter for buffered channels (i.e. queues). For unbuffered channels it is a non-issue.

Add a “reraise” method to RunFailedError

While having __cause__ set on RunFailedError helps produce a more useful traceback, it’s less helpful when handling the original error. To help facilitate this, we could add RunFailedError.reraise(). This method would enable the following pattern:

try:
    interp.run(script)
except RunFailedError as exc:
    try:
        exc.reraise()
    except MyException:
        ...

This would be made even simpler if there existed a __reraise__ protocol.

Support prioritization in channels

A simple example is queue.PriorityQueue in the stdlib.

Support inheriting settings (and more?)

Folks might find it useful, when creating a new subinterpreter, to be able to indicate that they would like some things “inherited” by the new interpreter. The mechanism could be a strict copy or it could be copy-on-write. The motivating example is with the warnings module (e.g. copy the filters).

The feature isn’t critical, nor would it be widely useful, so it can wait until there’s interest. Notably, both suggested solutions will require significant work, especially when it comes to complex objects and most especially for mutable containers of mutable complex objects.

Rejected Ideas

Explicit channel association

Interpreters are implicitly associated with channels upon recv() and send() calls. They are de-associated with release() calls. The alternative would be explicit methods. It would be either add_channel() and remove_channel() methods on Interpreter objects or something similar on channel objects.

In practice, this level of management shouldn’t be necessary for users. So adding more explicit support would only add clutter to the API.

Use pipes instead of channels

A pipe would be a simplex FIFO between exactly two interpreters. For most use cases this would be sufficient. It could potentially simplify the implementation as well. However, it isn’t a big step to supporting a many-to-many simplex FIFO via channels. Also, with pipes the API ends up being slightly more complicated, requiring naming the pipes.

Use queues instead of channels

Queues and buffered channels are almost the same thing. The main difference is that channels has a stronger relationship with context (i.e. the associated interpreter).

The name “Channel” was used instead of “Queue” to avoid confusion with the stdlib queue module.

Note that buffering in channels does complicate the blocking semantics of recv() and send(). Also, queues can be built on top of unbuffered channels.

“enumerate”

The list_all() function provides the list of all interpreters. In the threading module, which partly inspired the proposed API, the function is called enumerate(). The name is different here to avoid confusing Python users that are not already familiar with the threading API. For them “enumerate” is rather unclear, whereas “list_all” is clear.

Alternate solutions to prevent leaking exceptions across interpreters

In function calls, uncaught exceptions propagate to the calling frame. The same approach could be taken with run(). However, this would mean that exception objects would leak across the inter-interpreter boundary. Likewise, the frames in the traceback would potentially leak.

While that might not be a problem currently, it would be a problem once interpreters get better isolation relative to memory management (which is necessary to stop sharing the GIL between interpreters). We’ve resolved the semantics of how the exceptions propagate by raising a RunFailedError instead, for which __cause__ wraps a safe proxy for the original exception and traceback.

Rejected possible solutions:

• reproduce the exception and traceback in the original interpreter and raise that.

• raise a subclass of RunFailedError that proxies the original exception and traceback.

• raise RuntimeError instead of RunFailedError

• convert at the boundary (a la subprocess.CalledProcessError) (requires a cross-interpreter representation)

• support customization via Interpreter.excepthook (requires a cross-interpreter representation)

• wrap in a proxy at the boundary (including with support for something like err.raise() to propagate the traceback).

• return the exception (or its proxy) from run() instead of raising it

• return a result object (like subprocess does) [result-object] (unnecessary complexity?)

• throw the exception away and expect users to deal with unhandled exceptions explicitly in the script they pass to run() (they can pass error info out via channels); with threads you have to do something similar

Always associate each new interpreter with its own thread

As implemented in the C-API, a subinterpreter is not inherently tied to any thread. Furthermore, it will run in any existing thread, whether created by Python or not. You only have to activate one of its thread states (PyThreadState) in the thread first. This means that the same thread may run more than one interpreter (though obviously not at the same time).

The proposed module maintains this behavior. Subinterpreters are not tied to threads. Only calls to Interpreter.run() are. However, one of the key objectives of this PEP is to provide a more human- centric concurrency model. With that in mind, from a conceptual standpoint the module might be easier to understand if each subinterpreter were associated with its own thread.

That would mean interpreters.create() would create a new thread and Interpreter.run() would only execute in that thread (and nothing else would). The benefit is that users would not have to wrap Interpreter.run() calls in a new threading.Thread. Nor would they be in a position to accidentally pause the current interpreter (in the current thread) while their subinterpreter executes.

The idea is rejected because the benefit is small and the cost is high. The difference from the capability in the C-API would be potentially confusing. The implicit creation of threads is magical. The early creation of threads is potentially wasteful. The inability to run arbitrary interpreters in an existing thread would prevent some valid use cases, frustrating users. Tying interpreters to threads would require extra runtime modifications. It would also make the module’s implementation overly complicated. Finally, it might not even make the module easier to understand.

Only associate interpreters upon use

Associate interpreters with channel ends only once recv(), send(), etc. are called.

Doing this is potentially confusing and also can lead to unexpected races where a channel is auto-closed before it can be used in the original (creating) interpreter.

Implementation

The implementation of the PEP has 4 parts:

• the high-level module described in this PEP (mostly a light wrapper around a low-level C extension

• the low-level C extension module

• additions to the (“private”) C=API needed by the low-level module

• secondary fixes/changes in the CPython runtime that facilitate the low-level module (among other benefits)

These are at various levels of completion, with more done the lower you go:

• the high-level module has been, at best, roughly implemented. However, fully implementing it will be almost trivial.

• the low-level module is mostly complete. The bulk of the implementation was merged into master in December 2018 as the “_xxsubinterpreters” module (for the sake of testing subinterpreter functionality). Only 3 parts of the implementation remain: “send_wait()”, “send_buffer()”, and exception propagation. All three have been mostly finished, but were blocked by work related to ceval. That blocker is basically resolved now and finishing the low-level will not require extensive work.

• all necessary C-API work has been finished

• all anticipated work in the runtime has been finished

The implementation effort for PEP 554 is being tracked as part of a larger project aimed at improving multi-core support in CPython. [multi-core-project]

References

c-api

https://docs.python.org/3/c-api/init.html#sub-interpreter-support

CSP

https://en.wikipedia.org/wiki/Communicating_sequential_processes https://github.com/futurecore/python-csp

fifo

https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Pipe https://docs.python.org/3/library/multiprocessing.html#multiprocessing.Queue https://docs.python.org/3/library/queue.html#module-queue http://stackless.readthedocs.io/en/2.7-slp/library/stackless/channels.html https://golang.org/doc/effective_go.html#sharing http://www.jtolds.com/writing/2016/03/go-channels-are-bad-and-you-should-feel-bad/

caveats

https://docs.python.org/3/c-api/init.html#bugs-and-caveats

petr-c-ext

https://mail.python.org/pipermail/import-sig/2016-June/001062.html https://mail.python.org/pipermail/python-ideas/2016-April/039748.html

cryptography

https://github.com/pyca/cryptography/issues/2299

global-gc

http://bugs.python.org/issue24554

gilstate

https://bugs.python.org/issue10915 http://bugs.python.org/issue15751

global-atexit

https://bugs.python.org/issue6531

mp-conn

https://docs.python.org/3/library/multiprocessing.html#connection-objects

bug-rate

https://mail.python.org/pipermail/python-ideas/2017-September/047094.html

benefits

https://mail.python.org/pipermail/python-ideas/2017-September/047122.html

main-thread

https://mail.python.org/pipermail/python-ideas/2017-September/047144.html https://mail.python.org/pipermail/python-dev/2017-September/149566.html

reset_globals

https://mail.python.org/pipermail/python-dev/2017-September/149545.html

async

https://mail.python.org/pipermail/python-dev/2017-September/149420.html https://mail.python.org/pipermail/python-dev/2017-September/149585.html

result-object

https://mail.python.org/pipermail/python-dev/2017-September/149562.html

jython

https://mail.python.org/pipermail/python-ideas/2017-May/045771.html

multi-core-project

https://github.com/ericsnowcurrently/multi-core-python

cache-line-ping-pong

https://mail.python.org/archives/list/python-dev@python.org/message/3HVRFWHDMWPNR367GXBILZ4JJAUQ2STZ/

nathaniel-asyncio

https://mail.python.org/archives/list/python-dev@python.org/message/TUEAZNZHVJGGLL4OFD32OW6JJDKM6FAS/

extension-docs

https://docs.python.org/3/extending/index.html

Copyright

This document has been placed in the public domain.