Skip to content

bpo-35279: reduce default max_workers of ThreadPoolExecutor#13618

Merged
methane merged 2 commits intopython:masterfrom
methane:threadpool-defaultsize
May 28, 2019
Merged

bpo-35279: reduce default max_workers of ThreadPoolExecutor#13618
methane merged 2 commits intopython:masterfrom
methane:threadpool-defaultsize

Conversation

@methane
Copy link
Copy Markdown
Member

@methane methane commented May 28, 2019

@methane methane force-pushed the threadpool-defaultsize branch from d0d2ea3 to 4e7b131 Compare May 28, 2019 10:25
Copy link
Copy Markdown
Contributor

@asvetlov asvetlov left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM!


.. versionchanged:: 3.8
Default value of *max_workers* is changed to ``min(32, os.cpu_count() + 4)``.
This default value preserves at least 5 worker for I/O bound task. It utilize
Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Not a native English speaker but I think it should be "It utilizes" and "And it avoids" in the next sentence.

Copy link
Copy Markdown
Contributor

@csabella csabella May 28, 2019

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Also, at least 5 workers. And I'm not sure if it should be for I/O bound tasks or 'for the I/O bound task`.

.. versionchanged:: 3.8
Default value of *max_workers* is changed to ``min(32, os.cpu_count() + 4)``.
This default value preserves at least 5 worker for I/O bound task. It utilize
at most 32 CPU cores for CPU bound task which release GIL. And it avoid using
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

for CPU bound tasks and which release the GIL.

Default value of *max_workers* is changed to ``min(32, os.cpu_count() + 4)``.
This default value preserves at least 5 worker for I/O bound task. It utilize
at most 32 CPU cores for CPU bound task which release GIL. And it avoid using
very large resource implicitly on many core machines.
Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

s/resource/resources

@methane methane merged commit 9a7e5b1 into python:master May 28, 2019
@methane methane deleted the threadpool-defaultsize branch May 28, 2019 12:02
DinoV pushed a commit to DinoV/cpython that referenced this pull request Jan 14, 2020
It utilizes at most 32 CPU cores for CPU bound tasks which release the GIL.
And it avoids using very large resources implicitly on many-core machines.

ThreadPoolExecutor now reuses idle worker threads before starting
Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Where is the corresponding change in the code? For reusing the idle worker threads before starting max_workers worker threads?

Copy link
Copy Markdown
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

7 participants