Minor InterpreterPoolExecutor Issues

[ericsnowcurrently]

Bug report

Bug description:

• __main__ handling isn't quite right
• non-pickleable-but-shareable objects fail
• common code with test.support.interpreters should be shared

This has gh-132775 as a prerequisite.

CPython versions tested on:

CPython main branch

Operating systems tested on:

No response

Linked PRs

• gh-133485: Use interpreters.Interpreter in InterpreterPoolExecutor #133957
• [3.14] gh-133485: Use interpreters.Interpreter in InterpreterPoolExecutor (gh-133957) #135695

[henryiii]

I ran into this with the following example:

import argparse
import concurrent.futures
import random
import statistics
import time


EXECUTORS = {
    "thread": concurrent.futures.ThreadPoolExecutor,
    "process": concurrent.futures.ProcessPoolExecutor,
    "interpreter": concurrent.futures.InterpreterPoolExecutor,
}


def pi(trials: int) -> float:
    Ncirc = 0
    ran = random.Random()

    for _ in range(trials):
        x = ran.uniform(-1, 1)
        y = ran.uniform(-1, 1)

        test = x * x + y * y
        if test <= 1:
            Ncirc += 1

    return 4.0 * (Ncirc / trials)


def pi_in_threads(threads: int, trials: int, executor: str) -> float:
    if threads == 0:
        return pi(trials)
    Executor = EXECUTORS[executor]
    with Executor(max_workers=threads) as executor:
        return statistics.mean(executor.map(pi, [trials // threads] * threads))


def main() -> None:
    parser = argparse.ArgumentParser(color=True, suggest_on_error=True)
    parser.add_argument("--threads", type=int, default=0)
    parser.add_argument("--trials", type=int, default=10_000_000)
    parser.add_argument("--executor", choices=EXECUTORS, default="thread")
    args = parser.parse_args()

    start = time.monotonic()
    π = pi_in_threads(args.threads, args.trials, args.executor)
    stop = time.monotonic()

    print(f"{args.trials} trials, {π = }, {stop - start:.4} s")


if __name__ == "__main__":
    main()

It's unable to run with --executor interpreter because it hits oncurrent.futures.interpreter.ExecutionFailed: AttributeError: module '__main__' has no attribute 'pi'. The other two work. I'd assume this might be similar to how ProcessPoolExectuor handles __main__. Good to see there's already an issue, though, just dropping the script in case it's helpful.

[henryiii]

The syncobj parameter in the PEP isn't really explained, and it doesn't even seem to be present:

Traceback (most recent call last):
  File "/Users/henryschreiner/git/software/pybind11/concint.py", line 6, in <module>
    tasks = concurrent.interpreters.create_queue(syncobj=True)
TypeError: create() got an unexpected keyword argument 'syncobj'

I'm currently just sticking an integer into a queue, and in main interpreter I'm getting concurrent.interpreters._queues.UNBOUND out. I was wondering if this was related, but it's not even present AFAICT. I see unbounditems= in the code, which maybe is a new name? Is the plan to update the PEP, or just rely on CPython docs?

Is there some way to copy the item so it can be accessed after the interpreter is destroyed? This is my demo code:

Code:

import concurrent.interpreters
import concurrent.futures
import contextlib
import inspect

tasks = concurrent.interpreters.create_queue()

with (
    contextlib.closing(concurrent.interpreters.create()) as interp1,
    contextlib.closing(concurrent.interpreters.create()) as interp2
):

    with concurrent.futures.ThreadPoolExecutor() as pool:
        for interp in [interp1, interp2]:
            interp.prepare_main(tasks=tasks)
            pool.submit(interp.exec, inspect.cleandoc("""
                import time

                time.sleep(5)
                tasks.put(1)
                """
            ))

tasks.put(2)
print(tasks.get_nowait(), tasks.get_nowait(), tasks.get_nowait())

Indenting the final two lines fixes it, but curious if there's a way to get copies so I don't have to worry about the lifetimes, especially for things like integers.

[ericsnowcurrently]

It's unable to run with --executor interpreter because it hits oncurrent.futures.interpreter.ExecutionFailed: AttributeError: module 'main' has no attribute 'pi'. The other two work. I'd assume this might be similar to how ProcessPoolExectuor handles main. Good to see there's already an issue, though, just dropping the script in case it's helpful.

FYI, I just landed the key change here, which will be in beta 4. It should take care of the discrepancy between InterpreterPoolExecutor and the others.

The syncobj parameter in the PEP isn't really explained, and it doesn't even seem to be present:

Yeah, it's gone. I forgot to remove it from the PEP. I'll fix that.

Is there some way to copy the item so it can be accessed after the interpreter is destroyed? This is my demo code:

Objects belong to interpreters and an object cannot outlive the interpreter that created it.