849 Commits

Author SHA1 Message Date
Eric Snow
529a160be6
gh-117953: Share More Machinery Code Between Builtin and Dynamic Extensions (gh-118204)
This change will make some later changes simpler. It also brings more consistent behavior and lower maintenance costs.
2024-04-29 12:53:04 -06:00
Eric Snow
44f57a952e
gh-117953: Split Up _PyImport_LoadDynamicModuleWithSpec() (gh-118203)
Basically, I've turned most of _PyImport_LoadDynamicModuleWithSpec() into two new functions (_PyImport_GetModInitFunc() and _PyImport_RunModInitFunc()) and moved the rest of it out into _imp_create_dynamic_impl().  There shouldn't be any changes in behavior.

This change makes some future changes simpler.  This is particularly relevant to potentially calling each module init function in the main interpreter first.  Thus the critical part of the PR is the addition of _PyImport_RunModInitFunc(), which is strictly focused on running the init func and validating the result.  A later PR will take it a step farther by capturing error information rather than raising exceptions.

FWIW, this change also helps readers by clarifying a bit more about what happens when an extension/builtin module is imported.
2024-04-29 09:29:07 -06:00
Eric Snow
5865fa5f9b
gh-117953: Add Internal struct _Py_ext_module_loader_info (gh-118194)
This helps with a later change that splits up _PyImport_LoadDynamicModuleWithSpec().
2024-04-24 17:42:01 +00:00
Eric Snow
1acd249798
gh-117953: Let update_global_state_for_extension() Caller Decide If Singlephase or Not (gh-118193)
This change makes other upcoming changes simpler.
2024-04-24 16:28:35 +00:00
Eric Snow
af3c1d817d
gh-117953: Cleanups For fix_up_extension() in import.c (gh-118192)
These are cleanups I've pulled out of gh-118116.  Mostly, this change moves code around to align with some future changes and to improve clarity a little.  There is one very small change in behavior: we now add the module to the per-interpreter caches after updating the global state, rather than before.
2024-04-24 09:55:48 -06:00
Eric Snow
23950beff8
gh-117953: Small Cleanup of Extensions-Related Machinery Code (gh-118167)
This is a collection of very basic cleanups I've pulled out of gh-118116.  It is mostly renaming variables and moving a couple bits of code in functionally equivalent ways.
2024-04-23 08:25:50 -06:00
Sam Gross
25f6ff5d3e
gh-117649: Raise ImportError for unsupported modules in free-threaded build (#117651)
The free-threaded build does not currently support the combination of
single-phase init modules and non-isolated subinterpreters. Ensure that
`check_multi_interp_extensions` is always `True` for subinterpreters in
the free-threaded build so that importing these modules raises an
`ImportError`.
2024-04-11 15:00:54 -04:00
Mark Shannon
15309329b6
GH-108362: Incremental Cycle GC (GH-116206) 2024-03-20 08:54:42 +00:00
Victor Stinner
52d1477566
gh-110850: Rename internal PyTime C API functions (#115734)
Rename functions:

* _PyTime_GetSystemClock() => _PyTime_TimeUnchecked()
* _PyTime_GetPerfCounter() => _PyTime_PerfCounterUnchecked()
* _PyTime_GetMonotonicClock() => _PyTime_MonotonicUnchecked()
* _PyTime_GetSystemClockWithInfo() => _PyTime_TimeWithInfo()
* _PyTime_GetMonotonicClockWithInfo() => _PyTime_MonotonicWithInfo()
* _PyTime_GetMonotonicClockWithInfo() => _PyTime_MonotonicWithInfo()

Changes:

* Remove "typedef PyTime_t PyTime_t;" which was
  "typedef PyTime_t _PyTime_t;" before a previous rename.
* Update comments of "Unchecked" functions.
* Remove invalid PyTime_Time() comment.
2024-02-20 22:16:37 +00:00
Victor Stinner
d207c7cd5a
gh-110850: Cleanup pycore_time.h includes (#115724)
<pycore_time.h> include is no longer needed to get the PyTime_t type
in internal header files. This type is now provided by <Python.h>
include. Add <pycore_time.h> includes to C files instead.
2024-02-20 16:50:43 +00:00
Victor Stinner
9af80ec83d
gh-110850: Replace _PyTime_t with PyTime_t (#115719)
Run command:

sed -i -e 's!\<_PyTime_t\>!PyTime_t!g' $(find -name "*.c" -o -name "*.h")
2024-02-20 15:02:27 +00:00
Mark Shannon
8a3c499ffe
GH-108362: Revert "GH-108362: Incremental GC implementation (GH-108038)" (#115132)
Revert "GH-108362: Incremental GC implementation (GH-108038)"

This reverts commit 36518e69d74607e5f094ce55286188e4545a947d.
2024-02-07 12:38:34 +00:00
Mark Shannon
36518e69d7
GH-108362: Incremental GC implementation (GH-108038) 2024-02-05 18:28:51 +00:00
Nikita Sobolev
1ac1b2f953
gh-114685: Fix incorrect use of PyBUF_READ in import.c (GH-114686) 2024-01-29 11:37:06 +02:00
Sam Gross
cf6110ba13
gh-111924: Use PyMutex for Runtime-global Locks. (gh-112207)
This replaces some usages of PyThread_type_lock with PyMutex, which does not require memory allocation to initialize.

This simplifies some of the runtime initialization and is also one step towards avoiding changing the default raw memory allocator during initialize/finalization, which can be non-thread-safe in some circumstances.
2023-12-07 12:33:40 -07:00
Serhiy Storchaka
8660fb7fd7
gh-112660: Do not clear arbitrary errors on import (GH-112661)
Previously arbitrary errors could be cleared during formatting error
messages for ImportError or AttributeError for modules. Now all
unexpected errors are reported.
2023-12-07 12:19:43 +02:00
Victor Stinner
4f04172c92
gh-111262: Add PyDict_Pop() function (#112028)
_PyDict_Pop_KnownHash(): remove the default value and the return type
becomes an int.

Co-authored-by: Stefan Behnel <stefan_ml@behnel.de>
Co-authored-by: Antoine Pitrou <pitrou@free.fr>
2023-11-14 12:51:00 +00:00
Serhiy Storchaka
95365625f4
gh-111789: Simplify import.c by using PyDict_GetItemRef() (GH-111979) 2023-11-14 11:29:20 +02:00
Serhiy Storchaka
970e719a7a
gh-108082: Use PyErr_FormatUnraisable() (GH-111580)
Replace most of calls of _PyErr_WriteUnraisableMsg() and some
calls of PyErr_WriteUnraisable(NULL) with PyErr_FormatUnraisable().

Co-authored-by: Victor Stinner <vstinner@python.org>
2023-11-02 09:16:34 +00:00
scoder
a8a89fcd1f
gh-106320: Re-add some PyLong/PyDict C-API functions (GH-#111162)
* gh-106320: Re-add _PyLong_FromByteArray(), _PyLong_AsByteArray() and _PyLong_GCD() to the public header files since they are used by third-party packages and there is no efficient replacement.

See https://github.com/python/cpython/issues/111140
See https://github.com/python/cpython/issues/111139

* gh-111262: Re-add _PyDict_Pop() to have a C-API until a new public one is designed.
2023-10-25 11:33:48 +02:00
Sam Gross
cf6f23b0e3
gh-88402: Add new sysconfig variables on Windows (GH-110049)
Co-authored-by: Filipe Laíns <filipe.lains@gmail.com>
2023-10-04 22:50:29 +00:00
Victor Stinner
8b626a47ba
gh-110079: Remove extern "C" { ...} in C code (#110080) 2023-09-29 10:56:49 +02:00
Serhiy Storchaka
62c7015e89
gh-109521: Fix obscure cases handling in PyImport_GetImporter() (GH-109522)
PyImport_GetImporter() now sets RuntimeError if it fails to get sys.path_hooks
or sys.path_importer_cache or they are not list and dict correspondingly.

Previously it could return NULL without setting error in obscure cases,
crash or raise SystemError if these attributes have wrong type.
2023-09-23 09:39:24 +03:00
Serhiy Storchaka
add16f1a5e
gh-108511: Add C API functions which do not silently ignore errors (GH-109025)
Add the following functions:

* PyObject_HasAttrWithError()
* PyObject_HasAttrStringWithError()
* PyMapping_HasKeyWithError()
* PyMapping_HasKeyStringWithError()
2023-09-17 14:23:31 +03:00
Mark Shannon
15d4c9fabc
GH-108716: Turn off deep-freezing of code objects. (GH-108722) 2023-09-08 10:34:40 +01:00
Victor Stinner
26893016a7
gh-106320: Remove private _PyDict functions (#108449)
Move private functions to the internal C API (pycore_dict.h):

* _PyDictView_Intersect()
* _PyDictView_New()
* _PyDict_ContainsId()
* _PyDict_DelItemId()
* _PyDict_DelItem_KnownHash()
* _PyDict_GetItemIdWithError()
* _PyDict_GetItem_KnownHash()
* _PyDict_HasSplitTable()
* _PyDict_NewPresized()
* _PyDict_Next()
* _PyDict_Pop()
* _PyDict_SetItemId()
* _PyDict_SetItem_KnownHash()
* _PyDict_SizeOf()

No longer export most of these functions.

Move also the _PyDictViewObject structure to the internal C API.

Move dict_getitem_knownhash() function from _testcapi to the
_testinternalcapi extension. Update test_capi.test_dict for this
change.
2023-08-24 20:01:50 +00:00
Victor Stinner
4dc9f48930
gh-108308: Replace _PyDict_GetItemStringWithError() (#108372)
Replace _PyDict_GetItemStringWithError() calls with
PyDict_GetItemStringRef() which returns a strong reference to the
item.

Co-authored-by: Serhiy Storchaka <storchaka@gmail.com>
2023-08-23 22:59:00 +02:00
Victor Stinner
615f6e946d
gh-106320: Remove _PyDict_GetItemStringWithError() function (#108313)
Remove private _PyDict_GetItemStringWithError() function of the
public C API: the new PyDict_GetItemStringRef() can be used instead.

* Move private _PyDict_GetItemStringWithError() to the internal C API.
* _testcapi get_code_extra_index() uses PyDict_GetItemStringRef().
  Avoid using private functions in _testcapi which tests the public C
  API.
2023-08-22 18:17:25 +00:00
Eric Snow
017f047183
gh-107471: Fix Refleaks in test_import (gh-107569)
gh-107184 introduced a refleak in test_import.SubinterpImportTests (specifically test_singlephase_check_with_setting_and_override and test_single_init_extension_compat).  We fix it here by making sure _testsinglephase is removed from sys.modules whenever we clear the runtime's internal state for the module.

The underlying problem is strictly contained in the internal function _PyImport_ClearExtension() (AKA _testinternalcapi.clear_extension()), which is only used in tests.

(This also fixes an intermittent segfault introduced in the same place, in test_disallowed_reimport.)
2023-08-02 20:55:09 +00:00
Georg Brandl
f57b9fd2b6
no-issue: Fix typo in import.c (gh-107498) 2023-07-31 12:07:17 +00:00
Eric Snow
8ba4df91ae
gh-105699: Use a _Py_hashtable_t for the PyModuleDef Cache (gh-106974)
This fixes a crasher due to a race condition, triggered infrequently when two isolated (own GIL) subinterpreters simultaneously initialize their sys or builtins modules.  The crash happened due the combination of the "detached" thread state we were using and the "last holder" logic we use for the GIL.  It turns out it's tricky to use the same thread state for different threads.  Who could have guessed?

We solve the problem by eliminating the one object we were still sharing between interpreters.  We replace it with a low-level hashtable, using the "raw" allocator to avoid tying it to the main interpreter.

We also remove the accommodations for "detached" thread states, which were a dubious idea to start with.
2023-07-28 14:39:08 -06:00
Eric Snow
75c974f535
gh-104621: Check for Incompatible Extensions in import_find_extension() (gh-107184)
This fixes a bug where incompatible modules could still be imported if attempted multiple times.
2023-07-27 15:08:38 -06:00
Victor Stinner
188000ae4b
Remove unused internal _PyImport_GetModuleId() function (#107235) 2023-07-25 17:02:12 +00:00
Serhiy Storchaka
a293fa5915
gh-86493: Use PyModule_Add() instead of PyModule_AddObjectRef() (GH-106860) 2023-07-18 23:59:53 +03:00
Serhiy Storchaka
be1b968dc1
gh-106521: Remove _PyObject_LookupAttr() function (GH-106642) 2023-07-12 08:57:10 +03:00
Serhiy Storchaka
4bf43710d1
gh-106307: C API: Add PyMapping_GetOptionalItem() function (GH-106308)
Also add PyMapping_GetOptionalItemString() function.
2023-07-11 23:04:12 +03:00
Victor Stinner
18b1fdebe0
gh-106320: Remove _PyInterpreterState_Get() alias (#106321)
Replace calls to the (removed) slow _PyInterpreterState_Get() with
fast inlined _PyInterpreterState_GET() function.
2023-07-01 23:44:07 +00:00
Hood Chatham
e7bc8d1636
gh-106210 Remove Emscripten import trampoline (#106211)
It's no longer necessary.

Co-authored-by: Brett Cannon <brett@python.org>
2023-06-29 17:20:49 -07:00
Victor Stinner
00e75a3372
gh-106084: Remove old PyObject call aliases (#106085)
Remove old aliases which were kept backwards compatibility with
Python 3.8:

* _PyObject_CallMethodNoArgs()
* _PyObject_CallMethodOneArg()
* _PyObject_CallOneArg()
* _PyObject_FastCallDict()
* _PyObject_Vectorcall()
* _PyObject_VectorcallMethod()
* _PyVectorcall_Function()

Update code which used these aliases to use new names.
2023-06-26 08:08:12 +02:00
Victor Stinner
ee52158f20
gh-105927: PyImport_AddModule() uses _PyWeakref_GET_REF() (#106001)
It now raises an exception if sys.modules doesn't hold a strong
reference to the module.

Elaborate the comment explaining why a weak reference is used to
create a borrowed reference.
2023-06-23 01:23:08 +02:00
Victor Stinner
193a2b2eaa
gh-105922: Use PyImport_AddModuleRef() function (#105999)
Replace PyImport_AddModuleObject() + Py_XNewRef() with
PyImport_AddModuleRef() to get directly a strong reference.
2023-06-23 00:04:39 +02:00
chgnrdv
cd5280367a
gh-105979: Fix exception handling in unmarshal_frozen_code (Python/import.c) (#105980) 2023-06-22 21:30:19 +00:00
Victor Stinner
03f1a132ee
gh-105922: Add PyImport_AddModuleRef() function (#105923)
* Add tests on PyImport_AddModuleRef(), PyImport_AddModule() and
  PyImport_AddModuleObject().
* pythonrun.c: Replace Py_XNewRef(PyImport_AddModule(name)) with
  PyImport_AddModuleRef(name).
2023-06-20 08:48:14 +02:00
Eric Snow
b87d288275
gh-105699: Use a Thread-Local Variable for PKGCONTEXT (gh-105740)
This fixes a race during import. The existing _PyRuntimeState.imports.pkgcontext is shared between interpreters, and occasionally this would cause a crash when multiple interpreters were importing extensions modules at the same time.  To solve this we add a thread-local variable for the value.  We also leave the existing state (and infrequent race) in place for platforms that do not support thread-local variables.
2023-06-13 18:58:23 -06:00
Victor Stinner
3e525d2212
gh-105396: Deprecate PyImport_ImportModuleNoBlock() function (#105397)
Deprecate the PyImport_ImportModuleNoBlock() function which is just
an alias to PyImport_ImportModule() since Python 3.3.
2023-06-09 10:30:57 +02:00
Victor Stinner
ef300937c2
gh-92536: Remove PyUnicode_READY() calls (#105210)
Since Python 3.12, PyUnicode_READY() does nothing and always
returns 0.
2023-06-02 01:33:17 +02:00
Eric Snow
a9c6e0618f
gh-99113: Add Py_MOD_PER_INTERPRETER_GIL_SUPPORTED (gh-104205)
Here we are doing no more than adding the value for Py_mod_multiple_interpreters and using it for stdlib modules.  We will start checking for it in gh-104206 (once PyInterpreterState.ceval.own_gil is added in gh-104204).
2023-05-05 21:11:27 +00:00
Barry Warsaw
e1f14643dc
gh-98040: Remove just the imp module (#98573) 2023-04-28 16:17:58 -07:00
Eric Snow
d8627999d8
gh-100227: Add a Granular Lock for _PyRuntime.imports.extensions.dict (gh-103460)
The lock is unnecessary as long as there's a GIL, but completely
necessary with a per-interpreter GIL.
2023-04-24 21:09:35 -06:00
Eric Snow
19e4f757de
gh-100227: Only Use deepfreeze for the Main Interpreter (gh-103794)
Deep-frozen code objects are cannot be shared (currently) by
interpreters, due to how adaptive specialization can modify the
bytecodes. We work around this by only using the deep-frozen objects in
the main interpreter. This does incur a performance penalty for
subinterpreters, which we may be able to resolve later.
2023-04-24 21:48:05 +00:00