1berry/cpython
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

gh-84559: Remove the new multiprocessing warning, too disruptive. (#101551)

Browse Source 
This reverts the core of #100618 while leaving relevant documentation
improvements and minor refactorings in place.
This commit is contained in:
Gregory P. Smith 2023-02-03 15:20:46 -08:00 committed by GitHub
parent f6c53b80a1
commit d4c410f0f9
No known key found for this signature in database
GPG Key ID: 4AEE18F83AFDEB23
11 changed files with 27 additions and 189 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

16
Doc/library/concurrent.futures.rst
View File

@ -281,18 +281,18 @@ to a :class:`ProcessPoolExecutor` will result in deadlock.
Added the *initializer* and *initargs* arguments.
.. note::
The default :mod:`multiprocessing` start method
(see :ref:`multiprocessing-start-methods`) will change away from
*fork* in Python 3.14. Code that requires *fork* be used for their
:class:`ProcessPoolExecutor` should explicitly specify that by
passing a ``mp_context=multiprocessing.get_context("fork")``
parameter.
.. versionchanged:: 3.11
The *max_tasks_per_child* argument was added to allow users to
control the lifetime of workers in the pool.
.. versionchanged:: 3.12
The implicit use of the :mod:`multiprocessing` *fork* start method as a
platform default (see :ref:`multiprocessing-start-methods`) now raises a
:exc:`DeprecationWarning`. The default will change in Python 3.14.
Code that requires *fork* should explicitly specify that when creating
their :class:`ProcessPoolExecutor` by passing a
``mp_context=multiprocessing.get_context('fork')`` parameter.
.. _processpoolexecutor-example:
ProcessPoolExecutor Example

11
Doc/library/multiprocessing.rst
View File

@ -126,6 +126,11 @@ to start a process. These *start methods* are
Available on POSIX systems. Currently the default on POSIX except macOS.
.. note::
The default start method will change away from *fork* in Python 3.14.
Code that requires *fork* should explicitly specify that via
:func:`get_context` or :func:`set_start_method`.
*forkserver*
When the program starts and selects the *forkserver* start method,
a server process is spawned. From then on, whenever a new process
@ -138,11 +143,6 @@ to start a process. These *start methods* are
Available on POSIX platforms which support passing file descriptors
over Unix pipes such as Linux.
.. versionchanged:: 3.12
Implicit use of the *fork* start method as the default now raises a
:exc:`DeprecationWarning`. Code that requires it should explicitly
specify *fork* via :func:`get_context` or :func:`set_start_method`.
The default will change away from *fork* in 3.14.
.. versionchanged:: 3.8
@ -1107,6 +1107,7 @@ Miscellaneous
launched (before creating a :class:`Pool` or starting a :class:`Process`).
Only meaningful when using the ``'forkserver'`` start method.
See :ref:`multiprocessing-start-methods`.
.. versionadded:: 3.4

16
Doc/whatsnew/3.12.rst
View File

@ -440,12 +440,6 @@ Deprecated
warning at compile time. This field will be removed in Python 3.14.
(Contributed by Ramvikrams and Kumar Aditya in :gh:`101193`. PEP by Ken Jin.)
* Use of the implicit default ``'fork'`` start method for
:mod:`multiprocessing` and :class:`concurrent.futures.ProcessPoolExecutor`
now emits a :exc:`DeprecationWarning` on Linux and other non-macOS POSIX
systems. Avoid this by explicitly specifying a start method.
See :ref:`multiprocessing-start-methods`.
Pending Removal in Python 3.13
------------------------------
@ -510,9 +504,13 @@ Pending Removal in Python 3.14
* Testing the truth value of an :class:`xml.etree.ElementTree.Element`
is deprecated and will raise an exception in Python 3.14.
* The default :mod:`multiprocessing` start method will change to one of either
``'forkserver'`` or ``'spawn'`` on all platforms for which ``'fork'`` remains
the default per :gh:`84559`.
* The default :mod:`multiprocessing` start method will change to a safer one on
Linux, BSDs, and other non-macOS POSIX platforms where ``'fork'`` is currently
the default (:gh:`84559`). Adding a runtime warning about this was deemed too
disruptive as the majority of code is not expected to care. Use the
:func:`~multiprocessing.get_context` or
:func:`~multiprocessing.set_start_method` APIs to explicitly specify when
your code *requires* ``'fork'``. See :ref:`multiprocessing-start-methods`.
Pending Removal in Future Versions
----------------------------------

17
Lib/concurrent/futures/process.py
View File

@ -57,7 +57,6 @@ from functools import partial
import itertools
import sys
from traceback import format_exception
import warnings
_threads_wakeups = weakref.WeakKeyDictionary()
@ -651,22 +650,6 @@ class ProcessPoolExecutor(_base.Executor):
mp_context = mp.get_context("spawn")
else:
mp_context = mp.get_context()
if (mp_context.get_start_method() == "fork" and
mp_context == mp.context._default_context._default_context):
warnings.warn(
"The default multiprocessing start method will change "
"away from 'fork' in Python >= 3.14, per GH-84559. "
"ProcessPoolExecutor uses multiprocessing. "
"If your application requires the 'fork' multiprocessing "
"start method, explicitly specify that by passing a "
"mp_context= parameter. "
"The safest start method is 'spawn'.",
category=mp.context.DefaultForkDeprecationWarning,
stacklevel=2,
)
# Avoid the equivalent warning from multiprocessing itself via
# a non-default fork context.
mp_context = mp.get_context("fork")
self._mp_context = mp_context
# https://github.com/python/cpython/issues/90622

28
Lib/multiprocessing/context.py
View File

@ -23,9 +23,6 @@ class TimeoutError(ProcessError):
class AuthenticationError(ProcessError):
pass
class DefaultForkDeprecationWarning(DeprecationWarning):
pass
#
# Base type for contexts. Bound methods of an instance of this type are included in __all__ of __init__.py
#
@ -284,23 +281,6 @@ if sys.platform != 'win32':
from .popen_fork import Popen
return Popen(process_obj)
_warn_package_prefixes = (os.path.dirname(__file__),)
class _DeprecatedForkProcess(ForkProcess):
@classmethod
def _Popen(cls, process_obj):
import warnings
warnings.warn(
"The default multiprocessing start method will change "
"away from 'fork' in Python >= 3.14, per GH-84559. "
"Use multiprocessing.get_context(X) or .set_start_method(X) to "
"explicitly specify it when your application requires 'fork'. "
"The safest start method is 'spawn'.",
category=DefaultForkDeprecationWarning,
skip_file_prefixes=_warn_package_prefixes,
)
return super()._Popen(process_obj)
class SpawnProcess(process.BaseProcess):
_start_method = 'spawn'
@staticmethod
@ -324,9 +304,6 @@ if sys.platform != 'win32':
_name = 'fork'
Process = ForkProcess
class _DefaultForkContext(ForkContext):
Process = _DeprecatedForkProcess
class SpawnContext(BaseContext):
_name = 'spawn'
Process = SpawnProcess
@ -342,16 +319,13 @@ if sys.platform != 'win32':
'fork': ForkContext(),
'spawn': SpawnContext(),
'forkserver': ForkServerContext(),
# Remove None and _DefaultForkContext() when changing the default
# in 3.14 for https://github.com/python/cpython/issues/84559.
None: _DefaultForkContext(),
}
if sys.platform == 'darwin':
# bpo-33725: running arbitrary code after fork() is no longer reliable
# on macOS since macOS 10.14 (Mojave). Use spawn by default instead.
_default_context = DefaultContext(_concrete_contexts['spawn'])
else:
_default_context = DefaultContext(_concrete_contexts[None])
_default_context = DefaultContext(_concrete_contexts['fork'])
else:

5
Lib/test/_test_multiprocessing.py
View File

@ -4098,10 +4098,9 @@ class _TestSharedMemory(BaseTestCase):
def test_shared_memory_SharedMemoryManager_reuses_resource_tracker(self):
# bpo-36867: test that a SharedMemoryManager uses the
# same resource_tracker process as its parent.
cmd = f'''if 1:
cmd = '''if 1:
from multiprocessing.managers import SharedMemoryManager
from multiprocessing import set_start_method
set_start_method({multiprocessing.get_start_method()!r})
smm = SharedMemoryManager()
smm.start()

19
Lib/test/test_concurrent_futures.py
View File

@ -18,7 +18,6 @@ import sys
import threading
import time
import unittest
import warnings
import weakref
from pickle import PicklingError
@ -572,24 +571,6 @@ class ProcessPoolShutdownTest(ExecutorShutdownTest):
assert all([r == abs(v) for r, v in zip(res, range(-5, 5))])
@unittest.skipIf(mp.get_all_start_methods()[0] != "fork", "non-fork default.")
class ProcessPoolExecutorDefaultForkWarning(unittest.TestCase):
def test_fork_default_warns(self):
with self.assertWarns(mp.context.DefaultForkDeprecationWarning):
with futures.ProcessPoolExecutor(2):
pass
def test_explicit_fork_does_not_warn(self):
with warnings.catch_warnings(record=True) as ws:
warnings.simplefilter("ignore")
warnings.filterwarnings(
'always', category=mp.context.DefaultForkDeprecationWarning)
ctx = mp.get_context("fork") # Non-default fork context.
with futures.ProcessPoolExecutor(2, mp_context=ctx):
pass
self.assertEqual(len(ws), 0, msg=[str(x) for x in ws])
create_executor_tests(ProcessPoolShutdownTest,
executor_mixins=(ProcessPoolForkMixin,
ProcessPoolForkserverMixin,

5
Lib/test/test_logging.py
View File

@ -4759,9 +4759,8 @@ class LogRecordTest(BaseTest):
# In other processes, processName is correct when multiprocessing in imported,
# but it is (incorrectly) defaulted to 'MainProcess' otherwise (bpo-38762).
import multiprocessing
mp = multiprocessing.get_context('spawn')
parent_conn, child_conn = mp.Pipe()
p = mp.Process(
parent_conn, child_conn = multiprocessing.Pipe()
p = multiprocessing.Process(
target=self._extract_logrecord_process_name,
args=(2, LOG_MULTI_PROCESSING, child_conn,)
)

85
Lib/test/test_multiprocessing_defaults.py
View File

@ -1,85 +0,0 @@
"""Test default behavior of multiprocessing."""
from inspect import currentframe, getframeinfo
import multiprocessing
from multiprocessing.context import DefaultForkDeprecationWarning
import sys
from test.support import import_helper, threading_helper
import unittest
import warnings
# Skip tests if _multiprocessing wasn't built.
import_helper.import_module('_multiprocessing')
def do_nothing():
pass
# Process has the same API as Thread so this helper works.
join_process = threading_helper.join_thread
class DefaultWarningsTest(unittest.TestCase):
@unittest.skipIf(sys.platform in ('win32', 'darwin'),
'The default is not "fork" on Windows or macOS.')
def setUp(self):
self.assertEqual(multiprocessing.get_start_method(), 'fork')
self.assertIsInstance(multiprocessing.get_context(),
multiprocessing.context._DefaultForkContext)
def test_default_fork_start_method_warning_process(self):
with warnings.catch_warnings(record=True) as ws:
warnings.simplefilter('ignore')
warnings.filterwarnings('always', category=DefaultForkDeprecationWarning)
process = multiprocessing.Process(target=do_nothing)
process.start() # warning should point here.
join_process(process)
self.assertIsInstance(ws[0].message, DefaultForkDeprecationWarning)
self.assertIn(__file__, ws[0].filename)
self.assertEqual(getframeinfo(currentframe()).lineno-4, ws[0].lineno)
self.assertIn("'fork'", str(ws[0].message))
self.assertIn("get_context", str(ws[0].message))
self.assertEqual(len(ws), 1, msg=[str(x) for x in ws])
def test_default_fork_start_method_warning_pool(self):
with warnings.catch_warnings(record=True) as ws:
warnings.simplefilter('ignore')
warnings.filterwarnings('always', category=DefaultForkDeprecationWarning)
pool = multiprocessing.Pool(1) # warning should point here.
pool.terminate()
pool.join()
self.assertIsInstance(ws[0].message, DefaultForkDeprecationWarning)
self.assertIn(__file__, ws[0].filename)
self.assertEqual(getframeinfo(currentframe()).lineno-5, ws[0].lineno)
self.assertIn("'fork'", str(ws[0].message))
self.assertIn("get_context", str(ws[0].message))
self.assertEqual(len(ws), 1, msg=[str(x) for x in ws])
def test_default_fork_start_method_warning_manager(self):
with warnings.catch_warnings(record=True) as ws:
warnings.simplefilter('ignore')
warnings.filterwarnings('always', category=DefaultForkDeprecationWarning)
manager = multiprocessing.Manager() # warning should point here.
manager.shutdown()
self.assertIsInstance(ws[0].message, DefaultForkDeprecationWarning)
self.assertIn(__file__, ws[0].filename)
self.assertEqual(getframeinfo(currentframe()).lineno-4, ws[0].lineno)
self.assertIn("'fork'", str(ws[0].message))
self.assertIn("get_context", str(ws[0].message))
self.assertEqual(len(ws), 1, msg=[str(x) for x in ws])
def test_no_mp_warning_when_using_explicit_fork_context(self):
with warnings.catch_warnings(record=True) as ws:
warnings.simplefilter('ignore')
warnings.filterwarnings('always', category=DefaultForkDeprecationWarning)
fork_mp = multiprocessing.get_context('fork')
pool = fork_mp.Pool(1)
pool.terminate()
pool.join()
self.assertEqual(len(ws), 0, msg=[str(x) for x in ws])
if __name__ == '__main__':
unittest.main()

3
Lib/test/test_re.py
View File

@ -2431,8 +2431,7 @@ class ReTests(unittest.TestCase):
input_js = '''a(function() {
///////////////////////////////////////////////////////////////////
});'''
mp = multiprocessing.get_context('spawn')
p = mp.Process(target=pattern.sub, args=('', input_js))
p = multiprocessing.Process(target=pattern.sub, args=('', input_js))
p.start()
p.join(SHORT_TIMEOUT)
try:

11
Misc/NEWS.d/next/Library/2023-01-01-01-19-33.gh-issue-84559.zEjsEJ.rst
View File

@ -1,11 +0,0 @@
The :mod:`multiprocessing` module and
:class:`concurrent.futures.ProcessPoolExecutor` will emit a
:exc:`DeprecationWarning` on Linux and other non-macOS POSIX systems when
the default multiprocessing start method of ``'fork'`` is used implicitly
rather than being explicitly specified through a
:func:`multiprocessing.get_context` context.
This is in preparation for default start method to change in Python 3.14 to
a default that is safe for multithreaded applications.
Windows and macOS are unaffected as their default start method is ``spawn``.