1berry/cpython
1
0
Fork 0
Code Issues Packages Projects Releases Wiki Activity

gh-132661: Implement PEP 750 (#132662)

Browse Source 
Co-authored-by: Lysandros Nikolaou <lisandrosnik@gmail.com>
Co-authored-by: Bénédikt Tran <10796600+picnixz@users.noreply.github.com>
Co-authored-by: Adam Turner <9087854+AA-Turner@users.noreply.github.com>
Co-authored-by: Hugo van Kemenade <1324225+hugovk@users.noreply.github.com>
Co-authored-by: Wingy <git@wingysam.xyz>
Co-authored-by: Koudai Aono <koxudaxi@gmail.com>
Co-authored-by: Dave Peck <davepeck@gmail.com>
Co-authored-by: Terry Jan Reedy <tjreedy@udel.edu>
Co-authored-by: Paul Everitt <pauleveritt@me.com>
Co-authored-by: sobolevn <mail@sobolevn.me>
This commit is contained in:
Lysandros Nikolaou 2025-04-30 11:46:41 +02:00 committed by GitHub
parent 5ea9010e89
commit 60202609a2
No known key found for this signature in database
GPG Key ID: B5690EEEBB952194
81 changed files with 7716 additions and 3761 deletions
Show Stats Download Patch File Download Diff File Expand all files Collapse all files

6
.github/CODEOWNERS vendored
View File

@ -320,3 +320,9 @@ Lib/test/test__colorize.py @hugovk
# Fuzzing
Modules/_xxtestfuzz/ @ammaraskar
# t-strings
**/*interpolationobject* @lysnikolaou
**/*templateobject* @lysnikolaou
**/*templatelib* @lysnikolaou
**/*tstring* @lysnikolaou

35
Doc/library/token.rst
View File

@ -131,6 +131,41 @@ The token constants are:
The token string contains the closing quote(s).
.. data:: TSTRING_START
Token value used to indicate the beginning of a template string literal.
.. impl-detail::
The token string includes the prefix and the opening quote(s), but none
of the contents of the literal.
.. versionadded:: next
.. data:: TSTRING_MIDDLE
Token value used for literal text inside a template string literal
including format specifications.
.. impl-detail::
Replacement fields (that is, the non-literal parts of t-strings) use
the same tokens as other expressions, and are delimited by
:data:`LBRACE`, :data:`RBRACE`, :data:`EXCLAMATION` and :data:`COLON`
tokens.
.. versionadded:: next
.. data:: TSTRING_END
Token value used to indicate the end of a template string literal.
.. impl-detail::
The token string contains the closing quote(s).
.. versionadded:: next
.. data:: ENDMARKER
Token value that indicates the end of input.

71
Doc/whatsnew/3.14.rst
View File

@ -66,6 +66,7 @@ Summary -- release highlights
* :ref:`PEP 649: deferred evaluation of annotations <whatsnew314-pep649>`
* :ref:`PEP 741: Python Configuration C API <whatsnew314-pep741>`
* :ref:`PEP 750: Template Strings <whatsnew314-pep750>`
* :ref:`PEP 758: Allow except and except* expressions without parentheses <whatsnew314-pep758>`
* :ref:`PEP 761: Discontinuation of PGP signatures <whatsnew314-pep761>`
* :ref:`PEP 765: Disallow return/break/continue that exit a finally block <whatsnew314-pep765>`
@ -92,6 +93,76 @@ If you encounter :exc:`NameError`\s or pickling errors coming out of
New features
============
.. _whatsnew314-pep750:
PEP 750: Template Strings
-------------------------
Template string literals (t-strings) are a generalization of f-strings,
using a ``t`` in place of the ``f`` prefix. Instead of evaluating
to :class:`str`, t-strings evaluate to a new :class:`!string.templatelib.Template` type:
.. code-block:: python
from string.templatelib import Template
name = "World"
template: Template = t"Hello {name}"
The template can then be combined with functions that operate on the template's
structure to produce a :class:`str` or a string-like result.
For example, sanitizing input:
.. code-block:: python
evil = "<script>alert('evil')</script>"
template = t"<p>{evil}</p>"
assert html(template) == "<p>&lt;script&gt;alert('evil')&lt;/script&gt;</p>"
As another example, generating HTML attributes from data:
.. code-block:: python
attributes = {"src": "shrubbery.jpg", "alt": "looks nice"}
template = t"<img {attributes} />"
assert html(template) == '<img src="shrubbery.jpg" alt="looks nice" class="looks-nice" />'
Unlike f-strings, the ``html`` function has access to template attributes
containing the original information: static strings, interpolations, and values
from the original scope. Unlike existing templating approaches, t-strings build
from the well-known f-string syntax and rules. Template systems thus benefit
from Python tooling as they are much closer to the Python language, syntax,
scoping, and more.
Writing template handlers is straightforward:
.. code-block:: python
from string.templatelib import Template, Interpolation
def lower_upper(template: Template) -> str:
"""Render static parts lowercased and interpolations uppercased."""
parts: list[str] = []
for item in template:
if isinstance(item, Interpolation):
parts.append(str(item.value).upper())
else:
parts.append(item.lower())
return "".join(parts)
name = "world"
assert lower_upper(t"HELLO {name}") == "hello WORLD"
With this in place, developers can write template systems to sanitize SQL, make
safe shell operations, improve logging, tackle modern ideas in web development
(HTML, CSS, and so on), and implement lightweight, custom business DSLs.
See :pep:`750` for more details.
(Contributed by Jim Baker, Guido van Rossum, Paul Everitt, Koudai Aono,
Lysandros Nikolaou, Dave Peck, Adam Turner, Jelle Zijlstra, Bénédikt Tran,
and Pablo Galindo Salgado in :gh:`132661`.)
.. _whatsnew314-pep768:
PEP 768: Safe external debugger interface for CPython

3
Grammar/Tokens
View File

@ -62,6 +62,9 @@ SOFT_KEYWORD
FSTRING_START
FSTRING_MIDDLE
FSTRING_END
TSTRING_START
TSTRING_MIDDLE
TSTRING_END
COMMENT
NL
ERRORTOKEN

59
Grammar/python.gram
View File

@ -519,7 +519,7 @@ literal_pattern[pattern_ty]:
literal_expr[expr_ty]:
| signed_number !('+' | '-')
| complex_number
| strings
| &(STRING|FSTRING_START|TSTRING_START) strings
| 'None' { _PyAST_Constant(Py_None, NULL, EXTRA) }
| 'True' { _PyAST_Constant(Py_True, NULL, EXTRA) }
| 'False' { _PyAST_Constant(Py_False, NULL, EXTRA) }
@ -859,7 +859,7 @@ atom[expr_ty]:
| 'True' { _PyAST_Constant(Py_True, NULL, EXTRA) }
| 'False' { _PyAST_Constant(Py_False, NULL, EXTRA) }
| 'None' { _PyAST_Constant(Py_None, NULL, EXTRA) }
| &(STRING|FSTRING_START) strings
| &(STRING|FSTRING_START|TSTRING_START) strings
| NUMBER
| &'(' (tuple | group | genexp)
| &'[' (list | listcomp)
@ -935,7 +935,7 @@ fstring_middle[expr_ty]:
fstring_replacement_field[expr_ty]:
| '{' a=annotated_rhs debug_expr='='? conversion=[fstring_conversion] format=[fstring_full_format_spec] rbrace='}' {
_PyPegen_formatted_value(p, a, debug_expr, conversion, format, rbrace, EXTRA) }
| invalid_replacement_field
| invalid_fstring_replacement_field
fstring_conversion[ResultTokenWithMetadata*]:
| conv_token="!" conv=NAME { _PyPegen_check_fstring_conversion(p, conv_token, conv) }
fstring_full_format_spec[ResultTokenWithMetadata*]:
@ -946,8 +946,27 @@ fstring_format_spec[expr_ty]:
fstring[expr_ty]:
| a=FSTRING_START b=fstring_middle* c=FSTRING_END { _PyPegen_joined_str(p, a, (asdl_expr_seq*)b, c) }
tstring_format_spec_replacement_field[expr_ty]:
| '{' a=annotated_rhs debug_expr='='? conversion=[fstring_conversion] format=[tstring_full_format_spec] rbrace='}' {
_PyPegen_formatted_value(p, a, debug_expr, conversion, format, rbrace, EXTRA) }
| invalid_tstring_replacement_field
tstring_format_spec[expr_ty]:
| t=TSTRING_MIDDLE { _PyPegen_decoded_constant_from_token(p, t) }
| tstring_format_spec_replacement_field
tstring_full_format_spec[ResultTokenWithMetadata*]:
| colon=':' spec=tstring_format_spec* { _PyPegen_setup_full_format_spec(p, colon, (asdl_expr_seq *) spec, EXTRA) }
tstring_replacement_field[expr_ty]:
| '{' a=annotated_rhs debug_expr='='? conversion=[fstring_conversion] format=[tstring_full_format_spec] rbrace='}' {
_PyPegen_interpolation(p, a, debug_expr, conversion, format, rbrace, EXTRA) }
| invalid_tstring_replacement_field
tstring_middle[expr_ty]:
| tstring_replacement_field
| t=TSTRING_MIDDLE { _PyPegen_constant_from_token(p, t) }
tstring[expr_ty] (memo):
| a=TSTRING_START b=tstring_middle* c=TSTRING_END { _PyPegen_template_str(p, a, (asdl_expr_seq*)b, c) }
string[expr_ty]: s[Token*]=STRING { _PyPegen_constant_from_string(p, s) }
strings[expr_ty] (memo): a[asdl_expr_seq*]=(fstring|string)+ { _PyPegen_concatenate_strings(p, a, EXTRA) }
strings[expr_ty] (memo): a[asdl_expr_seq*]=(fstring|string|tstring)+ { _PyPegen_concatenate_strings(p, a, EXTRA) }
list[expr_ty]:
| '[' a=[star_named_expressions] ']' { _PyAST_List(a, Load, EXTRA) }
@ -1212,6 +1231,8 @@ invalid_expression:
RAISE_SYNTAX_ERROR_KNOWN_LOCATION (a, "expected expression before 'if', but statement is given") }
| a='lambda' [lambda_params] b=':' &FSTRING_MIDDLE {
RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "f-string: lambda expressions are not allowed without parentheses") }
| a='lambda' [lambda_params] b=':' &TSTRING_MIDDLE {
RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "t-string: lambda expressions are not allowed without parentheses") }
invalid_named_expression(memo):
| a=expression ':=' expression {
@ -1454,17 +1475,17 @@ invalid_starred_expression_unpacking:
invalid_starred_expression:
| '*' { RAISE_SYNTAX_ERROR("Invalid star expression") }
invalid_replacement_field:
invalid_fstring_replacement_field:
| '{' a='=' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: valid expression required before '='") }
| '{' a='!' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: valid expression required before '!'") }
| '{' a=':' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: valid expression required before ':'") }
| '{' a='}' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "f-string: valid expression required before '}'") }
| '{' !annotated_rhs { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting a valid expression after '{'")}
| '{' !annotated_rhs { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting a valid expression after '{'") }
| '{' annotated_rhs !('=' | '!' | ':' | '}') {
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting '=', or '!', or ':', or '}'") }
| '{' annotated_rhs '=' !('!' | ':' | '}') {
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting '!', or ':', or '}'") }
| '{' annotated_rhs '='? invalid_conversion_character
| '{' annotated_rhs '='? invalid_fstring_conversion_character
| '{' annotated_rhs '='? ['!' NAME] !(':' | '}') {
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting ':' or '}'") }
| '{' annotated_rhs '='? ['!' NAME] ':' fstring_format_spec* !'}' {
@ -1472,10 +1493,32 @@ invalid_replacement_field:
| '{' annotated_rhs '='? ['!' NAME] !'}' {
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: expecting '}'") }
invalid_conversion_character:
invalid_fstring_conversion_character:
| '!' &(':' | '}') { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: missing conversion character") }
| '!' !NAME { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("f-string: invalid conversion character") }
invalid_tstring_replacement_field:
| '{' a='=' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "t-string: valid expression required before '='") }
| '{' a='!' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "t-string: valid expression required before '!'") }
| '{' a=':' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "t-string: valid expression required before ':'") }
| '{' a='}' { RAISE_SYNTAX_ERROR_KNOWN_LOCATION(a, "t-string: valid expression required before '}'") }
| '{' !annotated_rhs { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting a valid expression after '{'") }
| '{' annotated_rhs !('=' | '!' | ':' | '}') {
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting '=', or '!', or ':', or '}'") }
| '{' annotated_rhs '=' !('!' | ':' | '}') {
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting '!', or ':', or '}'") }
| '{' annotated_rhs '='? invalid_tstring_conversion_character
| '{' annotated_rhs '='? ['!' NAME] !(':' | '}') {
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting ':' or '}'") }
| '{' annotated_rhs '='? ['!' NAME] ':' fstring_format_spec* !'}' {
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting '}', or format specs") }
| '{' annotated_rhs '='? ['!' NAME] !'}' {
PyErr_Occurred() ? NULL : RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: expecting '}'") }
invalid_tstring_conversion_character:
| '!' &(':' | '}') { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: missing conversion character") }
| '!' !NAME { RAISE_SYNTAX_ERROR_ON_NEXT_TOKEN("t-string: invalid conversion character") }
invalid_arithmetic:
| sum ('+'|'-'|'*'|'/'|'%'|'//'|'@') a='not' b=inversion { RAISE_SYNTAX_ERROR_KNOWN_RANGE(a, b, "'not' after an operator must be parenthesized") }
invalid_factor:

24
Include/internal/pycore_ast.h generated
View File

@ -361,9 +361,10 @@ enum _expr_kind {BoolOp_kind=1, NamedExpr_kind=2, BinOp_kind=3, UnaryOp_kind=4,
ListComp_kind=9, SetComp_kind=10, DictComp_kind=11,
GeneratorExp_kind=12, Await_kind=13, Yield_kind=14,
YieldFrom_kind=15, Compare_kind=16, Call_kind=17,
FormattedValue_kind=18, JoinedStr_kind=19, Constant_kind=20,
Attribute_kind=21, Subscript_kind=22, Starred_kind=23,
Name_kind=24, List_kind=25, Tuple_kind=26, Slice_kind=27};
FormattedValue_kind=18, Interpolation_kind=19,
JoinedStr_kind=20, TemplateStr_kind=21, Constant_kind=22,
Attribute_kind=23, Subscript_kind=24, Starred_kind=25,
Name_kind=26, List_kind=27, Tuple_kind=28, Slice_kind=29};
struct _expr {
enum _expr_kind kind;
union {
@ -459,10 +460,21 @@ struct _expr {
expr_ty format_spec;
} FormattedValue;
struct {
expr_ty value;
constant str;
int conversion;
expr_ty format_spec;
} Interpolation;
struct {
asdl_expr_seq *values;
} JoinedStr;
struct {
asdl_expr_seq *values;
} TemplateStr;
struct {
constant value;
string kind;
@ -820,8 +832,14 @@ expr_ty _PyAST_Call(expr_ty func, asdl_expr_seq * args, asdl_keyword_seq *
expr_ty _PyAST_FormattedValue(expr_ty value, int conversion, expr_ty
format_spec, int lineno, int col_offset, int
end_lineno, int end_col_offset, PyArena *arena);
expr_ty _PyAST_Interpolation(expr_ty value, constant str, int conversion,
expr_ty format_spec, int lineno, int col_offset,
int end_lineno, int end_col_offset, PyArena
*arena);
expr_ty _PyAST_JoinedStr(asdl_expr_seq * values, int lineno, int col_offset,
int end_lineno, int end_col_offset, PyArena *arena);
expr_ty _PyAST_TemplateStr(asdl_expr_seq * values, int lineno, int col_offset,
int end_lineno, int end_col_offset, PyArena *arena);
expr_ty _PyAST_Constant(constant value, string kind, int lineno, int
col_offset, int end_lineno, int end_col_offset, PyArena
*arena);

3
Include/internal/pycore_ast_state.h generated
View File

@ -75,6 +75,7 @@ struct ast_state {
PyObject *In_singleton;
PyObject *In_type;
PyObject *Interactive_type;
PyObject *Interpolation_type;
PyObject *Invert_singleton;
PyObject *Invert_type;
PyObject *IsNot_singleton;
@ -137,6 +138,7 @@ struct ast_state {
PyObject *Sub_singleton;
PyObject *Sub_type;
PyObject *Subscript_type;
PyObject *TemplateStr_type;
PyObject *TryStar_type;
PyObject *Try_type;
PyObject *Tuple_type;
@ -242,6 +244,7 @@ struct ast_state {
PyObject *slice;
PyObject *step;
PyObject *stmt_type;
PyObject *str;
PyObject *subject;
PyObject *tag;
PyObject *target;

2
Include/internal/pycore_global_objects_fini_generated.h generated
View File

@ -880,6 +880,7 @@ _PyStaticObjects_CheckRefcnt(PyInterpreterState *interp) {
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(consts));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(context));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(contravariant));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(conversion));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(cookie));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(copy));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(copyreg));
@ -937,6 +938,7 @@ _PyStaticObjects_CheckRefcnt(PyInterpreterState *interp) {
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(exception));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(existing_file_name));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(exp));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(expression));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(extend));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(extra_tokens));
_PyStaticObject_CheckRefcnt((PyObject *)&_Py_ID(facility));

2
Include/internal/pycore_global_strings.h
View File

@ -371,6 +371,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(consts)
STRUCT_FOR_ID(context)
STRUCT_FOR_ID(contravariant)
STRUCT_FOR_ID(conversion)
STRUCT_FOR_ID(cookie)
STRUCT_FOR_ID(copy)
STRUCT_FOR_ID(copyreg)
@ -428,6 +429,7 @@ struct _Py_global_strings {
STRUCT_FOR_ID(exception)
STRUCT_FOR_ID(existing_file_name)
STRUCT_FOR_ID(exp)
STRUCT_FOR_ID(expression)
STRUCT_FOR_ID(extend)
STRUCT_FOR_ID(extra_tokens)
STRUCT_FOR_ID(facility)

26
Include/internal/pycore_interpolation.h Normal file
View File

@ -0,0 +1,26 @@
#ifndef Py_INTERNAL_INTERPOLATION_H
#define Py_INTERNAL_INTERPOLATION_H
#ifdef __cplusplus
extern "C" {
#endif
#ifndef Py_BUILD_CORE
# error "this header requires Py_BUILD_CORE define"
#endif
extern PyTypeObject _PyInterpolation_Type;
#define _PyInterpolation_CheckExact(op) Py_IS_TYPE((op), &_PyInterpolation_Type)
PyAPI_FUNC(PyObject *) _PyInterpolation_Build(PyObject *value, PyObject *str,
int conversion, PyObject *format_spec);
extern PyStatus _PyInterpolation_InitTypes(PyInterpreterState *interp);
extern PyObject *_PyInterpolation_GetValueRef(PyObject *interpolation);
#ifdef __cplusplus
}
#endif
#endif

3
Include/internal/pycore_magic_number.h
View File

@ -275,6 +275,7 @@ Known values:
Python 3.14a6 3620 (Optimize bytecode for all/any/tuple called on a genexp)
Python 3.14a7 3621 (Optimize LOAD_FAST opcodes into LOAD_FAST_BORROW)
Python 3.14a7 3622 (Store annotations in different class dict keys)
Python 3.14a7 3623 (Add BUILD_INTERPOLATION & BUILD_TEMPLATE opcodes)
Python 3.15 will start with 3650
@ -287,7 +288,7 @@ PC/launcher.c must also be updated.
*/
#define PYC_MAGIC_NUMBER 3622
#define PYC_MAGIC_NUMBER 3623
/* This is equivalent to converting PYC_MAGIC_NUMBER to 2 bytes
(little-endian) and then appending b'\r\n'. */
#define PYC_MAGIC_NUMBER_TOKEN \

18
Include/internal/pycore_opcode_metadata.h generated
View File

@ -70,6 +70,8 @@ int _PyOpcode_num_popped(int opcode, int oparg) {
return 2;
case BINARY_SLICE:
return 3;
case BUILD_INTERPOLATION:
return 2 + (oparg & 1);
case BUILD_LIST:
return oparg;
case BUILD_MAP:
@ -80,6 +82,8 @@ int _PyOpcode_num_popped(int opcode, int oparg) {
return oparg;
case BUILD_STRING:
return oparg;
case BUILD_TEMPLATE:
return 2;
case BUILD_TUPLE:
return oparg;
case CACHE:
@ -551,6 +555,8 @@ int _PyOpcode_num_pushed(int opcode, int oparg) {
return 1;
case BINARY_SLICE:
return 1;
case BUILD_INTERPOLATION:
return 1;
case BUILD_LIST:
return 1;
case BUILD_MAP:
@ -561,6 +567,8 @@ int _PyOpcode_num_pushed(int opcode, int oparg) {
return 1;
case BUILD_STRING:
return 1;
case BUILD_TEMPLATE:
return 1;
case BUILD_TUPLE:
return 1;
case CACHE:
@ -1082,11 +1090,13 @@ const struct opcode_metadata _PyOpcode_opcode_metadata[267] = {
[BINARY_OP_SUBTRACT_FLOAT] = { true, INSTR_FMT_IXC0000, HAS_EXIT_FLAG | HAS_ERROR_FLAG },
[BINARY_OP_SUBTRACT_INT] = { true, INSTR_FMT_IXC0000, HAS_EXIT_FLAG | HAS_ERROR_FLAG | HAS_ESCAPES_FLAG },
[BINARY_SLICE] = { true, INSTR_FMT_IX, HAS_ERROR_FLAG | HAS_ESCAPES_FLAG },
[BUILD_INTERPOLATION] = { true, INSTR_FMT_IB, HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ESCAPES_FLAG },
[BUILD_LIST] = { true, INSTR_FMT_IB, HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG },
[BUILD_MAP] = { true, INSTR_FMT_IB, HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ESCAPES_FLAG },
[BUILD_SET] = { true, INSTR_FMT_IB, HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ESCAPES_FLAG },
[BUILD_SLICE] = { true, INSTR_FMT_IB, HAS_ARG_FLAG | HAS_ERROR_FLAG },
[BUILD_STRING] = { true, INSTR_FMT_IB, HAS_ARG_FLAG | HAS_ERROR_FLAG },
[BUILD_TEMPLATE] = { true, INSTR_FMT_IX, HAS_ERROR_FLAG | HAS_ESCAPES_FLAG },
[BUILD_TUPLE] = { true, INSTR_FMT_IB, HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG },
[CACHE] = { true, INSTR_FMT_IX, 0 },
[CALL] = { true, INSTR_FMT_IBC00, HAS_ARG_FLAG | HAS_EVAL_BREAK_FLAG | HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG },
@ -1330,11 +1340,13 @@ _PyOpcode_macro_expansion[256] = {
[BINARY_OP_SUBTRACT_FLOAT] = { .nuops = 3, .uops = { { _GUARD_TOS_FLOAT, OPARG_SIMPLE, 0 }, { _GUARD_NOS_FLOAT, OPARG_SIMPLE, 0 }, { _BINARY_OP_SUBTRACT_FLOAT, OPARG_SIMPLE, 5 } } },
[BINARY_OP_SUBTRACT_INT] = { .nuops = 3, .uops = { { _GUARD_TOS_INT, OPARG_SIMPLE, 0 }, { _GUARD_NOS_INT, OPARG_SIMPLE, 0 }, { _BINARY_OP_SUBTRACT_INT, OPARG_SIMPLE, 5 } } },
[BINARY_SLICE] = { .nuops = 1, .uops = { { _BINARY_SLICE, OPARG_SIMPLE, 0 } } },
[BUILD_INTERPOLATION] = { .nuops = 1, .uops = { { _BUILD_INTERPOLATION, OPARG_SIMPLE, 0 } } },
[BUILD_LIST] = { .nuops = 1, .uops = { { _BUILD_LIST, OPARG_SIMPLE, 0 } } },
[BUILD_MAP] = { .nuops = 1, .uops = { { _BUILD_MAP, OPARG_SIMPLE, 0 } } },
[BUILD_SET] = { .nuops = 1, .uops = { { _BUILD_SET, OPARG_SIMPLE, 0 } } },
[BUILD_SLICE] = { .nuops = 1, .uops = { { _BUILD_SLICE, OPARG_SIMPLE, 0 } } },
[BUILD_STRING] = { .nuops = 1, .uops = { { _BUILD_STRING, OPARG_SIMPLE, 0 } } },
[BUILD_TEMPLATE] = { .nuops = 1, .uops = { { _BUILD_TEMPLATE, OPARG_SIMPLE, 0 } } },
[BUILD_TUPLE] = { .nuops = 1, .uops = { { _BUILD_TUPLE, OPARG_SIMPLE, 0 } } },
[CALL_ALLOC_AND_ENTER_INIT] = { .nuops = 4, .uops = { { _CHECK_PEP_523, OPARG_SIMPLE, 1 }, { _CHECK_AND_ALLOCATE_OBJECT, 2, 1 }, { _CREATE_INIT_FRAME, OPARG_SIMPLE, 3 }, { _PUSH_FRAME, OPARG_SIMPLE, 3 } } },
[CALL_BOUND_METHOD_EXACT_ARGS] = { .nuops = 9, .uops = { { _CHECK_PEP_523, OPARG_SIMPLE, 1 }, { _CHECK_CALL_BOUND_METHOD_EXACT_ARGS, OPARG_SIMPLE, 1 }, { _INIT_CALL_BOUND_METHOD_EXACT_ARGS, OPARG_SIMPLE, 1 }, { _CHECK_FUNCTION_VERSION, 2, 1 }, { _CHECK_FUNCTION_EXACT_ARGS, OPARG_SIMPLE, 3 }, { _CHECK_STACK_SPACE, OPARG_SIMPLE, 3 }, { _INIT_CALL_PY_EXACT_ARGS, OPARG_SIMPLE, 3 }, { _SAVE_RETURN_OFFSET, OPARG_SAVE_RETURN_OFFSET, 3 }, { _PUSH_FRAME, OPARG_SIMPLE, 3 } } },
@ -1517,11 +1529,13 @@ const char *_PyOpcode_OpName[267] = {
[BINARY_OP_SUBTRACT_FLOAT] = "BINARY_OP_SUBTRACT_FLOAT",
[BINARY_OP_SUBTRACT_INT] = "BINARY_OP_SUBTRACT_INT",
[BINARY_SLICE] = "BINARY_SLICE",
[BUILD_INTERPOLATION] = "BUILD_INTERPOLATION",
[BUILD_LIST] = "BUILD_LIST",
[BUILD_MAP] = "BUILD_MAP",
[BUILD_SET] = "BUILD_SET",
[BUILD_SLICE] = "BUILD_SLICE",
[BUILD_STRING] = "BUILD_STRING",
[BUILD_TEMPLATE] = "BUILD_TEMPLATE",
[BUILD_TUPLE] = "BUILD_TUPLE",
[CACHE] = "CACHE",
[CALL] = "CALL",
@ -1782,11 +1796,13 @@ const uint8_t _PyOpcode_Deopt[256] = {
[BINARY_OP_SUBTRACT_FLOAT] = BINARY_OP,
[BINARY_OP_SUBTRACT_INT] = BINARY_OP,
[BINARY_SLICE] = BINARY_SLICE,
[BUILD_INTERPOLATION] = BUILD_INTERPOLATION,
[BUILD_LIST] = BUILD_LIST,
[BUILD_MAP] = BUILD_MAP,
[BUILD_SET] = BUILD_SET,
[BUILD_SLICE] = BUILD_SLICE,
[BUILD_STRING] = BUILD_STRING,
[BUILD_TEMPLATE] = BUILD_TEMPLATE,
[BUILD_TUPLE] = BUILD_TUPLE,
[CACHE] = CACHE,
[CALL] = CALL,
@ -1995,8 +2011,6 @@ const uint8_t _PyOpcode_Deopt[256] = {
#endif // NEED_OPCODE_METADATA
#define EXTRA_CASES \
case 119: \
case 120: \
case 121: \
case 122: \
case 123: \

2
Include/internal/pycore_runtime_init_generated.h generated
View File

@ -878,6 +878,7 @@ extern "C" {
INIT_ID(consts), \
INIT_ID(context), \
INIT_ID(contravariant), \
INIT_ID(conversion), \
INIT_ID(cookie), \
INIT_ID(copy), \
INIT_ID(copyreg), \
@ -935,6 +936,7 @@ extern "C" {
INIT_ID(exception), \
INIT_ID(existing_file_name), \
INIT_ID(exp), \
INIT_ID(expression), \
INIT_ID(extend), \
INIT_ID(extra_tokens), \
INIT_ID(facility), \

26
Include/internal/pycore_template.h Normal file
View File

@ -0,0 +1,26 @@
#ifndef Py_INTERNAL_TEMPLATE_H
#define Py_INTERNAL_TEMPLATE_H
#ifdef __cplusplus
extern "C" {
#endif
#ifndef Py_BUILD_CORE
# error "this header requires Py_BUILD_CORE define"
#endif
extern PyTypeObject _PyTemplate_Type;
extern PyTypeObject _PyTemplateIter_Type;
#define _PyTemplate_CheckExact(op) Py_IS_TYPE((op), &_PyTemplate_Type)
#define _PyTemplateIter_CheckExact(op) Py_IS_TYPE((op), &_PyTemplateIter_Type)
extern PyObject *_PyTemplate_Concat(PyObject *self, PyObject *other);
PyAPI_FUNC(PyObject *) _PyTemplate_Build(PyObject *strings, PyObject *interpolations);
#ifdef __cplusplus
}
#endif
#endif

14
Include/internal/pycore_token.h
View File

@ -75,10 +75,13 @@ extern "C" {
#define FSTRING_START 59
#define FSTRING_MIDDLE 60
#define FSTRING_END 61
#define COMMENT 62
#define NL 63
#define ERRORTOKEN 64
#define N_TOKENS 66
#define TSTRING_START 62
#define TSTRING_MIDDLE 63
#define TSTRING_END 64
#define COMMENT 65
#define NL 66
#define ERRORTOKEN 67
#define N_TOKENS 69
#define NT_OFFSET 256
/* Special definitions for cooperation with parser */
@ -91,7 +94,8 @@ extern "C" {
(x) == INDENT || \
(x) == DEDENT)
#define ISSTRINGLIT(x) ((x) == STRING || \
(x) == FSTRING_MIDDLE)
(x) == FSTRING_MIDDLE || \
(x) == TSTRING_MIDDLE)
// Export these 4 symbols for 'test_peg_generator'

8
Include/internal/pycore_unicodeobject_generated.h generated
View File

@ -1272,6 +1272,10 @@ _PyUnicode_InitStaticStrings(PyInterpreterState *interp) {
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));
assert(PyUnicode_GET_LENGTH(string) != 1);
string = &_Py_ID(conversion);
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));
assert(PyUnicode_GET_LENGTH(string) != 1);
string = &_Py_ID(cookie);
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));
@ -1500,6 +1504,10 @@ _PyUnicode_InitStaticStrings(PyInterpreterState *interp) {
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));
assert(PyUnicode_GET_LENGTH(string) != 1);
string = &_Py_ID(expression);
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));
assert(PyUnicode_GET_LENGTH(string) != 1);
string = &_Py_ID(extend);
_PyUnicode_InternStatic(interp, &string);
assert(_PyUnicode_CheckConsistency(string, 1));

2
Include/internal/pycore_uop_ids.h generated
View File

@ -28,11 +28,13 @@ extern "C" {
#define _BINARY_OP_SUBTRACT_FLOAT 316
#define _BINARY_OP_SUBTRACT_INT 317
#define _BINARY_SLICE 318
#define _BUILD_INTERPOLATION BUILD_INTERPOLATION
#define _BUILD_LIST BUILD_LIST
#define _BUILD_MAP BUILD_MAP
#define _BUILD_SET BUILD_SET
#define _BUILD_SLICE BUILD_SLICE
#define _BUILD_STRING BUILD_STRING
#define _BUILD_TEMPLATE BUILD_TEMPLATE
#define _BUILD_TUPLE BUILD_TUPLE
#define _CALL_BUILTIN_CLASS 319
#define _CALL_BUILTIN_FAST 320

8
Include/internal/pycore_uop_metadata.h generated
View File

@ -150,6 +150,8 @@ const uint16_t _PyUop_Flags[MAX_UOP_ID+1] = {
[_STORE_DEREF] = HAS_ARG_FLAG | HAS_FREE_FLAG | HAS_ESCAPES_FLAG,
[_COPY_FREE_VARS] = HAS_ARG_FLAG,
[_BUILD_STRING] = HAS_ARG_FLAG | HAS_ERROR_FLAG,
[_BUILD_INTERPOLATION] = HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ESCAPES_FLAG,
[_BUILD_TEMPLATE] = HAS_ERROR_FLAG | HAS_ESCAPES_FLAG,
[_BUILD_TUPLE] = HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG,
[_BUILD_LIST] = HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ERROR_NO_POP_FLAG | HAS_ESCAPES_FLAG,
[_LIST_EXTEND] = HAS_ARG_FLAG | HAS_ERROR_FLAG | HAS_ESCAPES_FLAG,
@ -330,11 +332,13 @@ const char *const _PyOpcode_uop_name[MAX_UOP_ID+1] = {
[_BINARY_OP_SUBTRACT_FLOAT] = "_BINARY_OP_SUBTRACT_FLOAT",
[_BINARY_OP_SUBTRACT_INT] = "_BINARY_OP_SUBTRACT_INT",
[_BINARY_SLICE] = "_BINARY_SLICE",
[_BUILD_INTERPOLATION] = "_BUILD_INTERPOLATION",
[_BUILD_LIST] = "_BUILD_LIST",
[_BUILD_MAP] = "_BUILD_MAP",
[_BUILD_SET] = "_BUILD_SET",
[_BUILD_SLICE] = "_BUILD_SLICE",
[_BUILD_STRING] = "_BUILD_STRING",
[_BUILD_TEMPLATE] = "_BUILD_TEMPLATE",
[_BUILD_TUPLE] = "_BUILD_TUPLE",
[_CALL_BUILTIN_CLASS] = "_CALL_BUILTIN_CLASS",
[_CALL_BUILTIN_FAST] = "_CALL_BUILTIN_FAST",
@ -862,6 +866,10 @@ int _PyUop_num_popped(int opcode, int oparg)
return 0;
case _BUILD_STRING:
return oparg;
case _BUILD_INTERPOLATION:
return 2 + (oparg & 1);
case _BUILD_TEMPLATE:
return 2;
case _BUILD_TUPLE:
return oparg;
case _BUILD_LIST:

234
Include/opcode_ids.h generated
View File

@ -12,123 +12,125 @@ extern "C" {
/* Instruction opcodes for compiled code */
#define CACHE 0
#define BINARY_SLICE 1
#define CALL_FUNCTION_EX 2
#define BUILD_TEMPLATE 2
#define BINARY_OP_INPLACE_ADD_UNICODE 3
#define CHECK_EG_MATCH 4
#define CHECK_EXC_MATCH 5
#define CLEANUP_THROW 6
#define DELETE_SUBSCR 7
#define END_FOR 8
#define END_SEND 9
#define EXIT_INIT_CHECK 10
#define FORMAT_SIMPLE 11
#define FORMAT_WITH_SPEC 12
#define GET_AITER 13
#define GET_ANEXT 14
#define GET_ITER 15
#define GET_LEN 16
#define CALL_FUNCTION_EX 4
#define CHECK_EG_MATCH 5
#define CHECK_EXC_MATCH 6
#define CLEANUP_THROW 7
#define DELETE_SUBSCR 8
#define END_FOR 9
#define END_SEND 10
#define EXIT_INIT_CHECK 11
#define FORMAT_SIMPLE 12
#define FORMAT_WITH_SPEC 13
#define GET_AITER 14
#define GET_ANEXT 15
#define GET_ITER 16
#define RESERVED 17
#define GET_YIELD_FROM_ITER 18
#define INTERPRETER_EXIT 19
#define LOAD_BUILD_CLASS 20
#define LOAD_LOCALS 21
#define MAKE_FUNCTION 22
#define MATCH_KEYS 23
#define MATCH_MAPPING 24
#define MATCH_SEQUENCE 25
#define NOP 26
#define NOT_TAKEN 27
#define POP_EXCEPT 28
#define POP_ITER 29
#define POP_TOP 30
#define PUSH_EXC_INFO 31
#define PUSH_NULL 32
#define RETURN_GENERATOR 33
#define RETURN_VALUE 34
#define SETUP_ANNOTATIONS 35
#define STORE_SLICE 36
#define STORE_SUBSCR 37
#define TO_BOOL 38
#define UNARY_INVERT 39
#define UNARY_NEGATIVE 40
#define UNARY_NOT 41
#define WITH_EXCEPT_START 42
#define BINARY_OP 43
#define BUILD_LIST 44
#define BUILD_MAP 45
#define BUILD_SET 46
#define BUILD_SLICE 47
#define BUILD_STRING 48
#define BUILD_TUPLE 49
#define CALL 50
#define CALL_INTRINSIC_1 51
#define CALL_INTRINSIC_2 52
#define CALL_KW 53
#define COMPARE_OP 54
#define CONTAINS_OP 55
#define CONVERT_VALUE 56
#define COPY 57
#define COPY_FREE_VARS 58
#define DELETE_ATTR 59
#define DELETE_DEREF 60
#define DELETE_FAST 61
#define DELETE_GLOBAL 62
#define DELETE_NAME 63
#define DICT_MERGE 64
#define DICT_UPDATE 65
#define END_ASYNC_FOR 66
#define EXTENDED_ARG 67
#define FOR_ITER 68
#define GET_AWAITABLE 69
#define IMPORT_FROM 70
#define IMPORT_NAME 71
#define IS_OP 72
#define JUMP_BACKWARD 73
#define JUMP_BACKWARD_NO_INTERRUPT 74
#define JUMP_FORWARD 75
#define LIST_APPEND 76
#define LIST_EXTEND 77
#define LOAD_ATTR 78
#define LOAD_COMMON_CONSTANT 79
#define LOAD_CONST 80
#define LOAD_DEREF 81
#define LOAD_FAST 82
#define LOAD_FAST_AND_CLEAR 83
#define LOAD_FAST_BORROW 84
#define LOAD_FAST_BORROW_LOAD_FAST_BORROW 85
#define LOAD_FAST_CHECK 86
#define LOAD_FAST_LOAD_FAST 87
#define LOAD_FROM_DICT_OR_DEREF 88
#define LOAD_FROM_DICT_OR_GLOBALS 89
#define LOAD_GLOBAL 90
#define LOAD_NAME 91
#define LOAD_SMALL_INT 92
#define LOAD_SPECIAL 93
#define LOAD_SUPER_ATTR 94
#define MAKE_CELL 95
#define MAP_ADD 96
#define MATCH_CLASS 97
#define POP_JUMP_IF_FALSE 98
#define POP_JUMP_IF_NONE 99
#define POP_JUMP_IF_NOT_NONE 100
#define POP_JUMP_IF_TRUE 101
#define RAISE_VARARGS 102
#define RERAISE 103
#define SEND 104
#define SET_ADD 105
#define SET_FUNCTION_ATTRIBUTE 106
#define SET_UPDATE 107
#define STORE_ATTR 108
#define STORE_DEREF 109
#define STORE_FAST 110
#define STORE_FAST_LOAD_FAST 111
#define STORE_FAST_STORE_FAST 112
#define STORE_GLOBAL 113
#define STORE_NAME 114
#define SWAP 115
#define UNPACK_EX 116
#define UNPACK_SEQUENCE 117
#define YIELD_VALUE 118
#define GET_LEN 18
#define GET_YIELD_FROM_ITER 19
#define INTERPRETER_EXIT 20
#define LOAD_BUILD_CLASS 21
#define LOAD_LOCALS 22
#define MAKE_FUNCTION 23
#define MATCH_KEYS 24
#define MATCH_MAPPING 25
#define MATCH_SEQUENCE 26
#define NOP 27
#define NOT_TAKEN 28
#define POP_EXCEPT 29
#define POP_ITER 30
#define POP_TOP 31
#define PUSH_EXC_INFO 32
#define PUSH_NULL 33
#define RETURN_GENERATOR 34
#define RETURN_VALUE 35
#define SETUP_ANNOTATIONS 36
#define STORE_SLICE 37
#define STORE_SUBSCR 38
#define TO_BOOL 39
#define UNARY_INVERT 40
#define UNARY_NEGATIVE 41
#define UNARY_NOT 42
#define WITH_EXCEPT_START 43
#define BINARY_OP 44
#define BUILD_INTERPOLATION 45
#define BUILD_LIST 46
#define BUILD_MAP 47
#define BUILD_SET 48
#define BUILD_SLICE 49
#define BUILD_STRING 50
#define BUILD_TUPLE 51
#define CALL 52
#define CALL_INTRINSIC_1 53
#define CALL_INTRINSIC_2 54
#define CALL_KW 55
#define COMPARE_OP 56
#define CONTAINS_OP 57
#define CONVERT_VALUE 58
#define COPY 59
#define COPY_FREE_VARS 60
#define DELETE_ATTR 61
#define DELETE_DEREF 62
#define DELETE_FAST 63
#define DELETE_GLOBAL 64
#define DELETE_NAME 65
#define DICT_MERGE 66
#define DICT_UPDATE 67
#define END_ASYNC_FOR 68
#define EXTENDED_ARG 69
#define FOR_ITER 70
#define GET_AWAITABLE 71
#define IMPORT_FROM 72
#define IMPORT_NAME 73
#define IS_OP 74
#define JUMP_BACKWARD 75
#define JUMP_BACKWARD_NO_INTERRUPT 76
#define JUMP_FORWARD 77
#define LIST_APPEND 78
#define LIST_EXTEND 79
#define LOAD_ATTR 80
#define LOAD_COMMON_CONSTANT 81
#define LOAD_CONST 82
#define LOAD_DEREF 83
#define LOAD_FAST 84
#define LOAD_FAST_AND_CLEAR 85
#define LOAD_FAST_BORROW 86
#define LOAD_FAST_BORROW_LOAD_FAST_BORROW 87
#define LOAD_FAST_CHECK 88
#define LOAD_FAST_LOAD_FAST 89
#define LOAD_FROM_DICT_OR_DEREF 90
#define LOAD_FROM_DICT_OR_GLOBALS 91
#define LOAD_GLOBAL 92
#define LOAD_NAME 93
#define LOAD_SMALL_INT 94
#define LOAD_SPECIAL 95
#define LOAD_SUPER_ATTR 96
#define MAKE_CELL 97
#define MAP_ADD 98
#define MATCH_CLASS 99
#define POP_JUMP_IF_FALSE 100
#define POP_JUMP_IF_NONE 101
#define POP_JUMP_IF_NOT_NONE 102
#define POP_JUMP_IF_TRUE 103
#define RAISE_VARARGS 104
#define RERAISE 105
#define SEND 106
#define SET_ADD 107
#define SET_FUNCTION_ATTRIBUTE 108
#define SET_UPDATE 109
#define STORE_ATTR 110
#define STORE_DEREF 111
#define STORE_FAST 112
#define STORE_FAST_LOAD_FAST 113
#define STORE_FAST_STORE_FAST 114
#define STORE_GLOBAL 115
#define STORE_NAME 116
#define SWAP 117
#define UNPACK_EX 118
#define UNPACK_SEQUENCE 119
#define YIELD_VALUE 120
#define RESUME 128
#define BINARY_OP_ADD_FLOAT 129
#define BINARY_OP_ADD_INT 130
@ -246,7 +248,7 @@ extern "C" {
#define SETUP_WITH 265
#define STORE_FAST_MAYBE_NULL 266
#define HAVE_ARGUMENT 42
#define HAVE_ARGUMENT 43
#define MIN_SPECIALIZED_OPCODE 129
#define MIN_INSTRUMENTED_OPCODE 234

90
Lib/_ast_unparse.py
View File

@ -573,21 +573,11 @@ class Unparser(NodeVisitor):
quote_type = quote_types[0]
self.write(f"{quote_type}{string}{quote_type}")
def visit_JoinedStr(self, node):
self.write("f")
fstring_parts = []
for value in node.values:
with self.buffered() as buffer:
self._write_fstring_inner(value)
fstring_parts.append(
("".join(buffer), isinstance(value, Constant))
)
new_fstring_parts = []
def _ftstring_helper(self, parts):
new_parts = []
quote_types = list(_ALL_QUOTES)
fallback_to_repr = False
for value, is_constant in fstring_parts:
for value, is_constant in parts:
if is_constant:
value, new_quote_types = self._str_literal_helper(
value,
@ -606,30 +596,68 @@ class Unparser(NodeVisitor):
new_quote_types = [q for q in quote_types if q not in value]
if new_quote_types:
quote_types = new_quote_types
new_fstring_parts.append(value)
new_parts.append(value)
if fallback_to_repr:
# If we weren't able to find a quote type that works for all parts
# of the JoinedStr, fallback to using repr and triple single quotes.
quote_types = ["'''"]
new_fstring_parts.clear()
for value, is_constant in fstring_parts:
new_parts.clear()
for value, is_constant in parts:
if is_constant:
value = repr('"' + value) # force repr to use single quotes
expected_prefix = "'\""
assert value.startswith(expected_prefix), repr(value)
value = value[len(expected_prefix):-1]
new_fstring_parts.append(value)
new_parts.append(value)
value = "".join(new_fstring_parts)
value = "".join(new_parts)
quote_type = quote_types[0]
self.write(f"{quote_type}{value}{quote_type}")
def _write_fstring_inner(self, node, is_format_spec=False):
def _write_ftstring(self, values, prefix):
self.write(prefix)
fstring_parts = []
for value in values:
with self.buffered() as buffer:
self._write_ftstring_inner(value)
fstring_parts.append(
("".join(buffer), isinstance(value, Constant))
)
self._ftstring_helper(fstring_parts)
def _tstring_helper(self, node):
last_idx = 0
for i, value in enumerate(node.values):
# This can happen if we have an implicit concat of a t-string
# with an f-string
if isinstance(value, FormattedValue):
if i > last_idx:
# Write t-string until here
self._write_ftstring(node.values[last_idx:i], "t")
self.write(" ")
# Write f-string with the current formatted value
self._write_ftstring([node.values[i]], "f")
if i + 1 < len(node.values):
# Only add a space if there are more values after this
self.write(" ")
last_idx = i + 1
if last_idx < len(node.values):
# Write t-string from last_idx to end
self._write_ftstring(node.values[last_idx:], "t")
def visit_JoinedStr(self, node):
self._write_ftstring(node.values, "f")
def visit_TemplateStr(self, node):
self._tstring_helper(node)
def _write_ftstring_inner(self, node, is_format_spec=False):
if isinstance(node, JoinedStr):
# for both the f-string itself, and format_spec
for value in node.values:
self._write_fstring_inner(value, is_format_spec=is_format_spec)
self._write_ftstring_inner(value, is_format_spec=is_format_spec)
elif isinstance(node, Constant) and isinstance(node.value, str):
value = node.value.replace("{", "{{").replace("}", "}}")
@ -641,17 +669,19 @@ class Unparser(NodeVisitor):
self.write(value)
elif isinstance(node, FormattedValue):
self.visit_FormattedValue(node)
elif isinstance(node, Interpolation):
self.visit_Interpolation(node)
else:
raise ValueError(f"Unexpected node inside JoinedStr, {node!r}")
def visit_FormattedValue(self, node):
def unparse_inner(inner):
unparser = type(self)()
unparser.set_precedence(_Precedence.TEST.next(), inner)
return unparser.visit(inner)
def _unparse_interpolation_value(self, inner):
unparser = type(self)()
unparser.set_precedence(_Precedence.TEST.next(), inner)
return unparser.visit(inner)
def _write_interpolation(self, node):
with self.delimit("{", "}"):
expr = unparse_inner(node.value)
expr = self._unparse_interpolation_value(node.value)
if expr.startswith("{"):
# Separate pair of opening brackets as "{ {"
self.write(" ")
@ -660,7 +690,13 @@ class Unparser(NodeVisitor):
self.write(f"!{chr(node.conversion)}")
if node.format_spec:
self.write(":")
self._write_fstring_inner(node.format_spec, is_format_spec=True)
self._write_ftstring_inner(node.format_spec, is_format_spec=True)
def visit_FormattedValue(self, node):
self._write_interpolation(node)
def visit_Interpolation(self, node):
self._write_interpolation(node)
def visit_Name(self, node):
self.write(node.id)

234
Lib/_opcode_metadata.py generated
View File

@ -215,121 +215,123 @@ opmap = {
'INSTRUMENTED_LINE': 254,
'ENTER_EXECUTOR': 255,
'BINARY_SLICE': 1,
'CALL_FUNCTION_EX': 2,
'CHECK_EG_MATCH': 4,
'CHECK_EXC_MATCH': 5,
'CLEANUP_THROW': 6,
'DELETE_SUBSCR': 7,
'END_FOR': 8,
'END_SEND': 9,
'EXIT_INIT_CHECK': 10,
'FORMAT_SIMPLE': 11,
'FORMAT_WITH_SPEC': 12,
'GET_AITER': 13,
'GET_ANEXT': 14,
'GET_ITER': 15,
'GET_LEN': 16,
'GET_YIELD_FROM_ITER': 18,
'INTERPRETER_EXIT': 19,
'LOAD_BUILD_CLASS': 20,
'LOAD_LOCALS': 21,
'MAKE_FUNCTION': 22,
'MATCH_KEYS': 23,
'MATCH_MAPPING': 24,
'MATCH_SEQUENCE': 25,
'NOP': 26,
'NOT_TAKEN': 27,
'POP_EXCEPT': 28,
'POP_ITER': 29,
'POP_TOP': 30,
'PUSH_EXC_INFO': 31,
'PUSH_NULL': 32,
'RETURN_GENERATOR': 33,
'RETURN_VALUE': 34,
'SETUP_ANNOTATIONS': 35,
'STORE_SLICE': 36,
'STORE_SUBSCR': 37,
'TO_BOOL': 38,
'UNARY_INVERT': 39,
'UNARY_NEGATIVE': 40,
'UNARY_NOT': 41,
'WITH_EXCEPT_START': 42,
'BINARY_OP': 43,
'BUILD_LIST': 44,
'BUILD_MAP': 45,
'BUILD_SET': 46,
'BUILD_SLICE': 47,
'BUILD_STRING': 48,
'BUILD_TUPLE': 49,
'CALL': 50,
'CALL_INTRINSIC_1': 51,
'CALL_INTRINSIC_2': 52,
'CALL_KW': 53,
'COMPARE_OP': 54,
'CONTAINS_OP': 55,
'CONVERT_VALUE': 56,
'COPY': 57,
'COPY_FREE_VARS': 58,
'DELETE_ATTR': 59,
'DELETE_DEREF': 60,
'DELETE_FAST': 61,
'DELETE_GLOBAL': 62,
'DELETE_NAME': 63,
'DICT_MERGE': 64,
'DICT_UPDATE': 65,
'END_ASYNC_FOR': 66,
'EXTENDED_ARG': 67,
'FOR_ITER': 68,
'GET_AWAITABLE': 69,
'IMPORT_FROM': 70,
'IMPORT_NAME': 71,
'IS_OP': 72,
'JUMP_BACKWARD': 73,
'JUMP_BACKWARD_NO_INTERRUPT': 74,
'JUMP_FORWARD': 75,
'LIST_APPEND': 76,
'LIST_EXTEND': 77,
'LOAD_ATTR': 78,
'LOAD_COMMON_CONSTANT': 79,
'LOAD_CONST': 80,
'LOAD_DEREF': 81,
'LOAD_FAST': 82,
'LOAD_FAST_AND_CLEAR': 83,
'LOAD_FAST_BORROW': 84,
'LOAD_FAST_BORROW_LOAD_FAST_BORROW': 85,
'LOAD_FAST_CHECK': 86,
'LOAD_FAST_LOAD_FAST': 87,
'LOAD_FROM_DICT_OR_DEREF': 88,
'LOAD_FROM_DICT_OR_GLOBALS': 89,
'LOAD_GLOBAL': 90,
'LOAD_NAME': 91,
'LOAD_SMALL_INT': 92,
'LOAD_SPECIAL': 93,
'LOAD_SUPER_ATTR': 94,
'MAKE_CELL': 95,
'MAP_ADD': 96,
'MATCH_CLASS': 97,
'POP_JUMP_IF_FALSE': 98,
'POP_JUMP_IF_NONE': 99,
'POP_JUMP_IF_NOT_NONE': 100,
'POP_JUMP_IF_TRUE': 101,
'RAISE_VARARGS': 102,
'RERAISE': 103,
'SEND': 104,
'SET_ADD': 105,
'SET_FUNCTION_ATTRIBUTE': 106,
'SET_UPDATE': 107,
'STORE_ATTR': 108,
'STORE_DEREF': 109,
'STORE_FAST': 110,
'STORE_FAST_LOAD_FAST': 111,
'STORE_FAST_STORE_FAST': 112,
'STORE_GLOBAL': 113,
'STORE_NAME': 114,
'SWAP': 115,
'UNPACK_EX': 116,
'UNPACK_SEQUENCE': 117,
'YIELD_VALUE': 118,
'BUILD_TEMPLATE': 2,
'CALL_FUNCTION_EX': 4,
'CHECK_EG_MATCH': 5,
'CHECK_EXC_MATCH': 6,
'CLEANUP_THROW': 7,
'DELETE_SUBSCR': 8,
'END_FOR': 9,
'END_SEND': 10,
'EXIT_INIT_CHECK': 11,
'FORMAT_SIMPLE': 12,
'FORMAT_WITH_SPEC': 13,
'GET_AITER': 14,
'GET_ANEXT': 15,
'GET_ITER': 16,
'GET_LEN': 18,
'GET_YIELD_FROM_ITER': 19,
'INTERPRETER_EXIT': 20,
'LOAD_BUILD_CLASS': 21,
'LOAD_LOCALS': 22,
'MAKE_FUNCTION': 23,
'MATCH_KEYS': 24,
'MATCH_MAPPING': 25,
'MATCH_SEQUENCE': 26,
'NOP': 27,
'NOT_TAKEN': 28,
'POP_EXCEPT': 29,
'POP_ITER': 30,
'POP_TOP': 31,
'PUSH_EXC_INFO': 32,
'PUSH_NULL': 33,
'RETURN_GENERATOR': 34,
'RETURN_VALUE': 35,
'SETUP_ANNOTATIONS': 36,
'STORE_SLICE': 37,
'STORE_SUBSCR': 38,
'TO_BOOL': 39,
'UNARY_INVERT': 40,
'UNARY_NEGATIVE': 41,
'UNARY_NOT': 42,
'WITH_EXCEPT_START': 43,
'BINARY_OP': 44,
'BUILD_INTERPOLATION': 45,
'BUILD_LIST': 46,
'BUILD_MAP': 47,
'BUILD_SET': 48,
'BUILD_SLICE': 49,
'BUILD_STRING': 50,
'BUILD_TUPLE': 51,
'CALL': 52,
'CALL_INTRINSIC_1': 53,
'CALL_INTRINSIC_2': 54,
'CALL_KW': 55,
'COMPARE_OP': 56,
'CONTAINS_OP': 57,
'CONVERT_VALUE': 58,
'COPY': 59,
'COPY_FREE_VARS': 60,
'DELETE_ATTR': 61,
'DELETE_DEREF': 62,
'DELETE_FAST': 63,
'DELETE_GLOBAL': 64,
'DELETE_NAME': 65,
'DICT_MERGE': 66,
'DICT_UPDATE': 67,
'END_ASYNC_FOR': 68,
'EXTENDED_ARG': 69,
'FOR_ITER': 70,
'GET_AWAITABLE': 71,
'IMPORT_FROM': 72,
'IMPORT_NAME': 73,
'IS_OP': 74,
'JUMP_BACKWARD': 75,
'JUMP_BACKWARD_NO_INTERRUPT': 76,
'JUMP_FORWARD': 77,
'LIST_APPEND': 78,
'LIST_EXTEND': 79,
'LOAD_ATTR': 80,
'LOAD_COMMON_CONSTANT': 81,
'LOAD_CONST': 82,
'LOAD_DEREF': 83,
'LOAD_FAST': 84,
'LOAD_FAST_AND_CLEAR': 85,
'LOAD_FAST_BORROW': 86,
'LOAD_FAST_BORROW_LOAD_FAST_BORROW': 87,
'LOAD_FAST_CHECK': 88,
'LOAD_FAST_LOAD_FAST': 89,
'LOAD_FROM_DICT_OR_DEREF': 90,
'LOAD_FROM_DICT_OR_GLOBALS': 91,
'LOAD_GLOBAL': 92,
'LOAD_NAME': 93,
'LOAD_SMALL_INT': 94,
'LOAD_SPECIAL': 95,
'LOAD_SUPER_ATTR': 96,
'MAKE_CELL': 97,
'MAP_ADD': 98,
'MATCH_CLASS': 99,
'POP_JUMP_IF_FALSE': 100,
'POP_JUMP_IF_NONE': 101,
'POP_JUMP_IF_NOT_NONE': 102,
'POP_JUMP_IF_TRUE': 103,
'RAISE_VARARGS': 104,
'RERAISE': 105,
'SEND': 106,
'SET_ADD': 107,
'SET_FUNCTION_ATTRIBUTE': 108,
'SET_UPDATE': 109,
'STORE_ATTR': 110,
'STORE_DEREF': 111,
'STORE_FAST': 112,
'STORE_FAST_LOAD_FAST': 113,
'STORE_FAST_STORE_FAST': 114,
'STORE_GLOBAL': 115,
'STORE_NAME': 116,
'SWAP': 117,
'UNPACK_EX': 118,
'UNPACK_SEQUENCE': 119,
'YIELD_VALUE': 120,
'INSTRUMENTED_END_FOR': 234,
'INSTRUMENTED_POP_ITER': 235,
'INSTRUMENTED_END_SEND': 236,
@ -363,5 +365,5 @@ opmap = {
'STORE_FAST_MAYBE_NULL': 266,
}
HAVE_ARGUMENT = 42
HAVE_ARGUMENT = 43
MIN_INSTRUMENTED_OPCODE = 234

0
Lib/string.py → Lib/string/__init__.py
View File

26
Lib/string/templatelib.py Normal file
View File

@ -0,0 +1,26 @@
"""Support for template string literals (t-strings)."""
__all__ = [
"Interpolation",
"Template",
]
t = t"{0}"
Template = type(t)
Interpolation = type(t.interpolations[0])
del t
def _template_unpickle(*args):
import itertools
if len(args) != 2:
raise ValueError('Template expects tuple of length 2 to unpickle')
strings, interpolations = args
parts = []
for string, interpolation in itertools.zip_longest(strings, interpolations):
if string is not None:
parts.append(string)
if interpolation is not None:
parts.append(interpolation)
return Template(*parts)

3
Lib/test/.ruff.toml
View File

@ -7,6 +7,9 @@ extend-exclude = [
# Non UTF-8 files
"encoded_modules/module_iso_8859_1.py",
"encoded_modules/module_koi8_r.py",
# SyntaxError because of t-strings
"test_tstring.py",
"test_string/test_templatelib.py",
# New grammar constructions may not yet be recognized by Ruff,
# and tests re-use the same names as only the grammar is being checked.
"test_grammar.py",

3
Lib/test/support/ast_helper.py
View File

@ -16,6 +16,9 @@ class ASTTestMixin:
self.fail(f"{type(a)!r} is not {type(b)!r}")
if isinstance(a, ast.AST):
for field in a._fields:
if isinstance(a, ast.Constant) and field == "kind":
# Skip the 'kind' field for ast.Constant
continue
value1 = getattr(a, field, missing)
value2 = getattr(b, field, missing)
# Singletons are equal by definition, so further

7
Lib/test/test_ast/data/ast_repr.txt
View File

@ -206,4 +206,9 @@ Module(body=[Expr(value=IfExp(test=Name(...), body=Call(...), orelse=Call(...)))
Module(body=[Expr(value=JoinedStr(values=[FormattedValue(...)]))], type_ignores=[])
Module(body=[Expr(value=JoinedStr(values=[FormattedValue(...)]))], type_ignores=[])
Module(body=[Expr(value=JoinedStr(values=[FormattedValue(...)]))], type_ignores=[])
Module(body=[Expr(value=JoinedStr(values=[Constant(...), ..., Constant(...)]))], type_ignores=[])
Module(body=[Expr(value=JoinedStr(values=[Constant(...), ..., Constant(...)]))], type_ignores=[])
Module(body=[Expr(value=TemplateStr(values=[Interpolation(...)]))], type_ignores=[])
Module(body=[Expr(value=TemplateStr(values=[Interpolation(...)]))], type_ignores=[])
Module(body=[Expr(value=TemplateStr(values=[Interpolation(...)]))], type_ignores=[])
Module(body=[Expr(value=TemplateStr(values=[Interpolation(...)]))], type_ignores=[])
Module(body=[Expr(value=TemplateStr(values=[Constant(...), ..., Constant(...)]))], type_ignores=[])

11
Lib/test/test_ast/snippets.py
View File

@ -364,6 +364,12 @@ eval_tests = [
"f'{a:.2f}'",
"f'{a!r}'",
"f'foo({a})'",
# TemplateStr and Interpolation
"t'{a}'",
"t'{a:.2f}'",
"t'{a!r}'",
"t'{a!r:.2f}'",
"t'foo({a})'",
]
@ -597,5 +603,10 @@ eval_results = [
('Expression', ('JoinedStr', (1, 0, 1, 10), [('FormattedValue', (1, 2, 1, 9), ('Name', (1, 3, 1, 4), 'a', ('Load',)), -1, ('JoinedStr', (1, 4, 1, 8), [('Constant', (1, 5, 1, 8), '.2f', None)]))])),
('Expression', ('JoinedStr', (1, 0, 1, 8), [('FormattedValue', (1, 2, 1, 7), ('Name', (1, 3, 1, 4), 'a', ('Load',)), 114, None)])),
('Expression', ('JoinedStr', (1, 0, 1, 11), [('Constant', (1, 2, 1, 6), 'foo(', None), ('FormattedValue', (1, 6, 1, 9), ('Name', (1, 7, 1, 8), 'a', ('Load',)), -1, None), ('Constant', (1, 9, 1, 10), ')', None)])),
('Expression', ('TemplateStr', (1, 0, 1, 6), [('Interpolation', (1, 2, 1, 5), ('Name', (1, 3, 1, 4), 'a', ('Load',)), 'a', -1, None)])),
('Expression', ('TemplateStr', (1, 0, 1, 10), [('Interpolation', (1, 2, 1, 9), ('Name', (1, 3, 1, 4), 'a', ('Load',)), 'a', -1, ('JoinedStr', (1, 4, 1, 8), [('Constant', (1, 5, 1, 8), '.2f', None)]))])),
('Expression', ('TemplateStr', (1, 0, 1, 8), [('Interpolation', (1, 2, 1, 7), ('Name', (1, 3, 1, 4), 'a', ('Load',)), 'a', 114, None)])),
('Expression', ('TemplateStr', (1, 0, 1, 12), [('Interpolation', (1, 2, 1, 11), ('Name', (1, 3, 1, 4), 'a', ('Load',)), 'a', 114, ('JoinedStr', (1, 6, 1, 10), [('Constant', (1, 7, 1, 10), '.2f', None)]))])),
('Expression', ('TemplateStr', (1, 0, 1, 11), [('Constant', (1, 2, 1, 6), 'foo(', None), ('Interpolation', (1, 6, 1, 9), ('Name', (1, 7, 1, 8), 'a', ('Load',)), 'a', -1, None), ('Constant', (1, 9, 1, 10), ')', None)])),
]
main()

19
Lib/test/test_ast/test_ast.py
View File

@ -880,6 +880,25 @@ class AST_Tests(unittest.TestCase):
for src in srcs:
ast.parse(src)
def test_tstring(self):
# Test AST structure for simple t-string
tree = ast.parse('t"Hello"')
self.assertIsInstance(tree.body[0].value, ast.TemplateStr)
self.assertIsInstance(tree.body[0].value.values[0], ast.Constant)
# Test AST for t-string with interpolation
tree = ast.parse('t"Hello {name}"')
self.assertIsInstance(tree.body[0].value, ast.TemplateStr)
self.assertIsInstance(tree.body[0].value.values[0], ast.Constant)
self.assertIsInstance(tree.body[0].value.values[1], ast.Interpolation)
# Test AST for implicit concat of t-string with f-string
tree = ast.parse('t"Hello {name}" f"{name}"')
self.assertIsInstance(tree.body[0].value, ast.TemplateStr)
self.assertIsInstance(tree.body[0].value.values[0], ast.Constant)
self.assertIsInstance(tree.body[0].value.values[1], ast.Interpolation)
self.assertIsInstance(tree.body[0].value.values[2], ast.FormattedValue)
class CopyTests(unittest.TestCase):
"""Test copying and pickling AST nodes."""

1
Lib/test/test_fstring.py
View File

@ -1358,7 +1358,6 @@ x = (
self.assertAllRaise(SyntaxError, "f-string: expecting '}'",
["f'{3!'",
"f'{3!s'",
"f'{3!g'",
])
self.assertAllRaise(SyntaxError, 'f-string: missing conversion character',

6
Lib/test/test_grammar.py
View File

@ -1507,6 +1507,8 @@ class GrammarTests(unittest.TestCase):
check('[None (3, 4)]')
check('[True (3, 4)]')
check('[... (3, 4)]')
check('[t"{x}" (3, 4)]')
check('[t"x={x}" (3, 4)]')
msg=r'is not subscriptable; perhaps you missed a comma\?'
check('[{1, 2} [i, j]]')
@ -1529,6 +1531,8 @@ class GrammarTests(unittest.TestCase):
check('[f"x={x}" [i, j]]')
check('["abc" [i, j]]')
check('[b"abc" [i, j]]')
check('[t"{x}" [i, j]]')
check('[t"x={x}" [i, j]]')
msg=r'indices must be integers or slices, not tuple;'
check('[[1, 2] [3, 4]]')
@ -1549,6 +1553,8 @@ class GrammarTests(unittest.TestCase):
check('[[1, 2] [f"{x}"]]')
check('[[1, 2] [f"x={x}"]]')
check('[[1, 2] ["abc"]]')
check('[[1, 2] [t"{x}"]]')
check('[[1, 2] [t"x={x}"]]')
msg=r'indices must be integers or slices, not'
check('[[1, 2] [b"abc"]]')
check('[[1, 2] [12.3]]')

5
Lib/test/test_string/__init__.py Normal file
View File

@ -0,0 +1,5 @@
import os
from test.support import load_package_tests
def load_tests(*args):
return load_package_tests(os.path.dirname(__file__), *args)

55
Lib/test/test_string/_support.py Normal file
View File

@ -0,0 +1,55 @@
import unittest
from string.templatelib import Interpolation
class TStringBaseCase:
def assertTStringEqual(self, t, strings, interpolations):
"""Test template string literal equality.
The *strings* argument must be a tuple of strings equal to *t.strings*.
The *interpolations* argument must be a sequence of tuples which are
compared against *t.interpolations*. Each tuple consists of
(value, expression, conversion, format_spec), though the final two
items may be omitted, and are assumed to be None and '' respectively.
"""
self.assertEqual(t.strings, strings)
self.assertEqual(len(t.interpolations), len(interpolations))
for i, exp in zip(t.interpolations, interpolations, strict=True):
if len(exp) == 4:
actual = (i.value, i.expression, i.conversion, i.format_spec)
self.assertEqual(actual, exp)
continue
if len(exp) == 3:
self.assertEqual((i.value, i.expression, i.conversion), exp)
self.assertEqual(i.format_spec, '')
continue
self.assertEqual((i.value, i.expression), exp)
self.assertEqual(i.format_spec, '')
self.assertIsNone(i.conversion)
def convert(value, conversion):
if conversion == "a":
return ascii(value)
elif conversion == "r":
return repr(value)
elif conversion == "s":
return str(value)
return value
def fstring(template):
parts = []
for item in template:
match item:
case str() as s:
parts.append(s)
case Interpolation(value, _, conversion, format_spec):
value = convert(value, conversion)
value = format(value, format_spec)
parts.append(value)
return "".join(parts)

0
Lib/test/test_string.py → Lib/test/test_string/test_string.py
View File

122
Lib/test/test_string/test_templatelib.py Normal file
View File

@ -0,0 +1,122 @@
import pickle
import unittest
from string.templatelib import Template, Interpolation
from test.test_string._support import TStringBaseCase, fstring
class TestTemplate(unittest.TestCase, TStringBaseCase):
def test_common(self):
self.assertEqual(type(t'').__name__, 'Template')
self.assertEqual(type(t'').__qualname__, 'Template')
self.assertEqual(type(t'').__module__, 'string.templatelib')
a = 'a'
i = t'{a}'.interpolations[0]
self.assertEqual(type(i).__name__, 'Interpolation')
self.assertEqual(type(i).__qualname__, 'Interpolation')
self.assertEqual(type(i).__module__, 'string.templatelib')
def test_basic_creation(self):
# Simple t-string creation
t = t'Hello, world'
self.assertIsInstance(t, Template)
self.assertTStringEqual(t, ('Hello, world',), ())
self.assertEqual(fstring(t), 'Hello, world')
# Empty t-string
t = t''
self.assertTStringEqual(t, ('',), ())
self.assertEqual(fstring(t), '')
# Multi-line t-string
t = t"""Hello,
world"""
self.assertEqual(t.strings, ('Hello,\nworld',))
self.assertEqual(len(t.interpolations), 0)
self.assertEqual(fstring(t), 'Hello,\nworld')
def test_creation_interleaving(self):
# Should add strings on either side
t = Template(Interpolation('Maria', 'name', None, ''))
self.assertTStringEqual(t, ('', ''), [('Maria', 'name')])
self.assertEqual(fstring(t), 'Maria')
# Should prepend empty string
t = Template(Interpolation('Maria', 'name', None, ''), ' is my name')
self.assertTStringEqual(t, ('', ' is my name'), [('Maria', 'name')])
self.assertEqual(fstring(t), 'Maria is my name')
# Should append empty string
t = Template('Hello, ', Interpolation('Maria', 'name', None, ''))
self.assertTStringEqual(t, ('Hello, ', ''), [('Maria', 'name')])
self.assertEqual(fstring(t), 'Hello, Maria')
# Should concatenate strings
t = Template('Hello', ', ', Interpolation('Maria', 'name', None, ''),
'!')
self.assertTStringEqual(t, ('Hello, ', '!'), [('Maria', 'name')])
self.assertEqual(fstring(t), 'Hello, Maria!')
# Should add strings on either side and in between
t = Template(Interpolation('Maria', 'name', None, ''),
Interpolation('Python', 'language', None, ''))
self.assertTStringEqual(
t, ('', '', ''), [('Maria', 'name'), ('Python', 'language')]
)
self.assertEqual(fstring(t), 'MariaPython')
def test_template_values(self):
t = t'Hello, world'
self.assertEqual(t.values, ())
name = "Lys"
t = t'Hello, {name}'
self.assertEqual(t.values, ("Lys",))
country = "GR"
age = 0
t = t'Hello, {name}, {age} from {country}'
self.assertEqual(t.values, ("Lys", 0, "GR"))
def test_pickle_template(self):
user = 'test'
for template in (
t'',
t"No values",
t'With inter {user}',
t'With ! {user!r}',
t'With format {1 / 0.3:.2f}',
Template(),
Template('a'),
Template(Interpolation('Nikita', 'name', None, '')),
Template('a', Interpolation('Nikita', 'name', 'r', '')),
):
for proto in range(pickle.HIGHEST_PROTOCOL + 1):
with self.subTest(proto=proto, template=template):
pickled = pickle.dumps(template, protocol=proto)
unpickled = pickle.loads(pickled)
self.assertEqual(unpickled.values, template.values)
self.assertEqual(fstring(unpickled), fstring(template))
def test_pickle_interpolation(self):
for interpolation in (
Interpolation('Nikita', 'name', None, ''),
Interpolation('Nikita', 'name', 'r', ''),
Interpolation(1/3, 'x', None, '.2f'),
):
for proto in range(pickle.HIGHEST_PROTOCOL + 1):
with self.subTest(proto=proto, interpolation=interpolation):
pickled = pickle.dumps(interpolation, protocol=proto)
unpickled = pickle.loads(pickled)
self.assertEqual(unpickled.value, interpolation.value)
self.assertEqual(unpickled.expression, interpolation.expression)
self.assertEqual(unpickled.conversion, interpolation.conversion)
self.assertEqual(unpickled.format_spec, interpolation.format_spec)
if __name__ == '__main__':
unittest.main()

8
Lib/test/test_syntax.py
View File

@ -1877,6 +1877,14 @@ SyntaxError: cannot assign to f-string expression here. Maybe you meant '==' ins
Traceback (most recent call last):
SyntaxError: cannot assign to f-string expression here. Maybe you meant '==' instead of '='?
>>> t'{x}' = 42
Traceback (most recent call last):
SyntaxError: cannot assign to t-string expression here. Maybe you meant '==' instead of '='?
>>> t'{x}-{y}' = 42
Traceback (most recent call last):
SyntaxError: cannot assign to t-string expression here. Maybe you meant '==' instead of '='?
>>> (x, y, z=3, d, e)
Traceback (most recent call last):
SyntaxError: invalid syntax. Maybe you meant '==' or ':=' instead of '='?

313
Lib/test/test_tstring.py Normal file
View File

@ -0,0 +1,313 @@
import unittest
from test.test_string._support import TStringBaseCase, fstring
class TestTString(unittest.TestCase, TStringBaseCase):
def test_string_representation(self):
# Test __repr__
t = t"Hello"
self.assertEqual(repr(t), "Template(strings=('Hello',), interpolations=())")
name = "Python"
t = t"Hello, {name}"
self.assertEqual(repr(t),
"Template(strings=('Hello, ', ''), "
"interpolations=(Interpolation('Python', 'name', None, ''),))"
)
def test_interpolation_basics(self):
# Test basic interpolation
name = "Python"
t = t"Hello, {name}"
self.assertTStringEqual(t, ("Hello, ", ""), [(name, "name")])
self.assertEqual(fstring(t), "Hello, Python")
# Multiple interpolations
first = "Python"
last = "Developer"
t = t"{first} {last}"
self.assertTStringEqual(
t, ("", " ", ""), [(first, 'first'), (last, 'last')]
)
self.assertEqual(fstring(t), "Python Developer")
# Interpolation with expressions
a = 10
b = 20
t = t"Sum: {a + b}"
self.assertTStringEqual(t, ("Sum: ", ""), [(a + b, "a + b")])
self.assertEqual(fstring(t), "Sum: 30")
# Interpolation with function
def square(x):
return x * x
t = t"Square: {square(5)}"
self.assertTStringEqual(
t, ("Square: ", ""), [(square(5), "square(5)")]
)
self.assertEqual(fstring(t), "Square: 25")
# Test attribute access in expressions
class Person:
def __init__(self, name):
self.name = name
def upper(self):
return self.name.upper()
person = Person("Alice")
t = t"Name: {person.name}"
self.assertTStringEqual(
t, ("Name: ", ""), [(person.name, "person.name")]
)
self.assertEqual(fstring(t), "Name: Alice")
# Test method calls
t = t"Name: {person.upper()}"
self.assertTStringEqual(
t, ("Name: ", ""), [(person.upper(), "person.upper()")]
)
self.assertEqual(fstring(t), "Name: ALICE")
# Test dictionary access
data = {"name": "Bob", "age": 30}
t = t"Name: {data['name']}, Age: {data['age']}"
self.assertTStringEqual(
t, ("Name: ", ", Age: ", ""),
[(data["name"], "data['name']"), (data["age"], "data['age']")],
)
self.assertEqual(fstring(t), "Name: Bob, Age: 30")
def test_format_specifiers(self):
# Test basic format specifiers
value = 3.14159
t = t"Pi: {value:.2f}"
self.assertTStringEqual(
t, ("Pi: ", ""), [(value, "value", None, ".2f")]
)
self.assertEqual(fstring(t), "Pi: 3.14")
def test_conversions(self):
# Test !s conversion (str)
obj = object()
t = t"Object: {obj!s}"
self.assertTStringEqual(t, ("Object: ", ""), [(obj, "obj", "s")])
self.assertEqual(fstring(t), f"Object: {str(obj)}")
# Test !r conversion (repr)
t = t"Data: {obj!r}"
self.assertTStringEqual(t, ("Data: ", ""), [(obj, "obj", "r")])
self.assertEqual(fstring(t), f"Data: {repr(obj)}")
# Test !a conversion (ascii)
text = "Café"
t = t"ASCII: {text!a}"
self.assertTStringEqual(t, ("ASCII: ", ""), [(text, "text", "a")])
self.assertEqual(fstring(t), f"ASCII: {ascii(text)}")
# Test !z conversion (error)
num = 1
with self.assertRaises(SyntaxError):
eval("t'{num!z}'")
def test_debug_specifier(self):
# Test debug specifier
value = 42
t = t"Value: {value=}"
self.assertTStringEqual(
t, ("Value: value=", ""), [(value, "value", "r")]
)
self.assertEqual(fstring(t), "Value: value=42")
# Test debug specifier with format (conversion default to !r)
t = t"Value: {value=:.2f}"
self.assertTStringEqual(
t, ("Value: value=", ""), [(value, "value", None, ".2f")]
)
self.assertEqual(fstring(t), "Value: value=42.00")
# Test debug specifier with conversion
t = t"Value: {value=!s}"
self.assertTStringEqual(
t, ("Value: value=", ""), [(value, "value", "s")]
)
# Test white space in debug specifier
t = t"Value: {value = }"
self.assertTStringEqual(
t, ("Value: value = ", ""), [(value, "value", "r")]
)
self.assertEqual(fstring(t), "Value: value = 42")
def test_raw_tstrings(self):
path = r"C:\Users"
t = rt"{path}\Documents"
self.assertTStringEqual(t, ("", r"\Documents"), [(path, "path")])
self.assertEqual(fstring(t), r"C:\Users\Documents")
# Test alternative prefix
t = tr"{path}\Documents"
self.assertTStringEqual(t, ("", r"\Documents"), [(path, "path")])
def test_template_concatenation(self):
# Test template + template
t1 = t"Hello, "
t2 = t"world"
combined = t1 + t2
self.assertTStringEqual(combined, ("Hello, world",), ())
self.assertEqual(fstring(combined), "Hello, world")
# Test template + string
t1 = t"Hello"
combined = t1 + ", world"
self.assertTStringEqual(combined, ("Hello, world",), ())
self.assertEqual(fstring(combined), "Hello, world")
# Test template + template with interpolation
name = "Python"
t1 = t"Hello, "
t2 = t"{name}"
combined = t1 + t2
self.assertTStringEqual(combined, ("Hello, ", ""), [(name, "name")])
self.assertEqual(fstring(combined), "Hello, Python")
# Test string + template
t = "Hello, " + t"{name}"
self.assertTStringEqual(t, ("Hello, ", ""), [(name, "name")])
self.assertEqual(fstring(t), "Hello, Python")
def test_nested_templates(self):
# Test a template inside another template expression
name = "Python"
inner = t"{name}"
t = t"Language: {inner}"
t_interp = t.interpolations[0]
self.assertEqual(t.strings, ("Language: ", ""))
self.assertEqual(t_interp.value.strings, ("", ""))
self.assertEqual(t_interp.value.interpolations[0].value, name)
self.assertEqual(t_interp.value.interpolations[0].expression, "name")
self.assertEqual(t_interp.value.interpolations[0].conversion, None)
self.assertEqual(t_interp.value.interpolations[0].format_spec, "")
self.assertEqual(t_interp.expression, "inner")
self.assertEqual(t_interp.conversion, None)
self.assertEqual(t_interp.format_spec, "")
def test_syntax_errors(self):
for case, err in (
("t'", "unterminated t-string literal"),
("t'''", "unterminated triple-quoted t-string literal"),
("t''''", "unterminated triple-quoted t-string literal"),
("t'{", "'{' was never closed"),
("t'{'", "t-string: expecting '}'"),
("t'{a'", "t-string: expecting '}'"),
("t'}'", "t-string: single '}' is not allowed"),
("t'{}'", "t-string: valid expression required before '}'"),
("t'{=x}'", "t-string: valid expression required before '='"),
("t'{!x}'", "t-string: valid expression required before '!'"),
("t'{:x}'", "t-string: valid expression required before ':'"),
("t'{x;y}'", "t-string: expecting '=', or '!', or ':', or '}'"),
("t'{x=y}'", "t-string: expecting '!', or ':', or '}'"),
("t'{x!s!}'", "t-string: expecting ':' or '}'"),
("t'{x!s:'", "t-string: expecting '}', or format specs"),
("t'{x!}'", "t-string: missing conversion character"),
("t'{x=!}'", "t-string: missing conversion character"),
("t'{x!z}'", "t-string: invalid conversion character 'z': "
"expected 's', 'r', or 'a'"),
("t'{lambda:1}'", "t-string: lambda expressions are not allowed "
"without parentheses"),
("t'{x:{;}}'", "t-string: expecting a valid expression after '{'"),
):
with self.subTest(case), self.assertRaisesRegex(SyntaxError, err):
eval(case)
def test_runtime_errors(self):
# Test missing variables
with self.assertRaises(NameError):
eval("t'Hello, {name}'")
def test_literal_concatenation(self):
# Test concatenation of t-string literals
t = t"Hello, " t"world"
self.assertTStringEqual(t, ("Hello, world",), ())
self.assertEqual(fstring(t), "Hello, world")
# Test concatenation with interpolation
name = "Python"
t = t"Hello, " t"{name}"
self.assertTStringEqual(t, ("Hello, ", ""), [(name, "name")])
self.assertEqual(fstring(t), "Hello, Python")
# Test concatenation with string literal
name = "Python"
t = t"Hello, {name}" "and welcome!"
self.assertTStringEqual(
t, ("Hello, ", "and welcome!"), [(name, "name")]
)
self.assertEqual(fstring(t), "Hello, Pythonand welcome!")
# Test concatenation with Unicode literal
name = "Python"
t = t"Hello, {name}" u"and welcome!"
self.assertTStringEqual(
t, ("Hello, ", "and welcome!"), [(name, "name")]
)
self.assertEqual(fstring(t), "Hello, Pythonand welcome!")
# Test concatenation with f-string literal
tab = '\t'
t = t"Tab: {tab}. " f"f-tab: {tab}."
self.assertTStringEqual(t, ("Tab: ", ". f-tab: \t."), [(tab, "tab")])
self.assertEqual(fstring(t), "Tab: \t. f-tab: \t.")
# Test concatenation with raw string literal
tab = '\t'
t = t"Tab: {tab}. " r"Raw tab: \t."
self.assertTStringEqual(
t, ("Tab: ", r". Raw tab: \t."), [(tab, "tab")]
)
self.assertEqual(fstring(t), "Tab: \t. Raw tab: \\t.")
# Test concatenation with raw f-string literal
tab = '\t'
t = t"Tab: {tab}. " rf"f-tab: {tab}. Raw tab: \t."
self.assertTStringEqual(
t, ("Tab: ", ". f-tab: \t. Raw tab: \\t."), [(tab, "tab")]
)
self.assertEqual(fstring(t), "Tab: \t. f-tab: \t. Raw tab: \\t.")
what = 't'
expected_msg = 'cannot mix bytes and nonbytes literals'
for case in (
"t'{what}-string literal' b'bytes literal'",
"t'{what}-string literal' br'raw bytes literal'",
):
with self.assertRaisesRegex(SyntaxError, expected_msg):
eval(case)
def test_triple_quoted(self):
# Test triple-quoted t-strings
t = t"""
Hello,
world
"""
self.assertTStringEqual(
t, ("\n Hello,\n world\n ",), ()
)
self.assertEqual(fstring(t), "\n Hello,\n world\n ")
# Test triple-quoted with interpolation
name = "Python"
t = t"""
Hello,
{name}
"""
self.assertTStringEqual(
t, ("\n Hello,\n ", "\n "), [(name, "name")]
)
self.assertEqual(fstring(t), "\n Hello,\n Python\n ")
if __name__ == '__main__':
unittest.main()

11
Lib/test/test_unparse.py
View File

@ -202,6 +202,15 @@ class UnparseTestCase(ASTTestCase):
self.check_ast_roundtrip('f" something { my_dict["key"] } something else "')
self.check_ast_roundtrip('f"{f"{f"{f"{f"{f"{1+1}"}"}"}"}"}"')
def test_tstrings(self):
self.check_ast_roundtrip("t'foo'")
self.check_ast_roundtrip("t'foo {bar}'")
self.check_ast_roundtrip("t'foo {bar!s:.2f}'")
self.check_ast_roundtrip("t'foo {bar}' f'{bar}'")
self.check_ast_roundtrip("f'{bar}' t'foo {bar}'")
self.check_ast_roundtrip("t'foo {bar}' fr'\\hello {bar}'")
self.check_ast_roundtrip("t'foo {bar}' u'bar'")
def test_strings(self):
self.check_ast_roundtrip("u'foo'")
self.check_ast_roundtrip("r'foo'")
@ -918,7 +927,7 @@ class DirectoryTestCase(ASTTestCase):
run_always_files = {"test_grammar.py", "test_syntax.py", "test_compile.py",
"test_ast.py", "test_asdl_parser.py", "test_fstring.py",
"test_patma.py", "test_type_alias.py", "test_type_params.py",
"test_tokenize.py"}
"test_tokenize.py", "test_tstring.py"}
_files_to_test = None

13
Lib/token.py generated
View File

@ -66,12 +66,15 @@ SOFT_KEYWORD = 58
FSTRING_START = 59
FSTRING_MIDDLE = 60
FSTRING_END = 61
COMMENT = 62
NL = 63
TSTRING_START = 62
TSTRING_MIDDLE = 63
TSTRING_END = 64
COMMENT = 65
NL = 66
# These aren't used by the C tokenizer but are needed for tokenize.py
ERRORTOKEN = 64
ENCODING = 65
N_TOKENS = 66
ERRORTOKEN = 67
ENCODING = 68
N_TOKENS = 69
# Special definitions for cooperation with parser
NT_OFFSET = 256

4
Lib/tokenize.py
View File

@ -251,7 +251,7 @@ class Untokenizer:
self.tokens.append(indent)
self.prev_col = len(indent)
startline = False
elif tok_type == FSTRING_MIDDLE:
elif tok_type in {FSTRING_MIDDLE, TSTRING_MIDDLE}:
if '{' in token or '}' in token:
token = self.escape_brackets(token)
last_line = token.splitlines()[-1]
@ -308,7 +308,7 @@ class Untokenizer:
elif startline and indents:
toks_append(indents[-1])
startline = False
elif toknum == FSTRING_MIDDLE:
elif toknum in {FSTRING_MIDDLE, TSTRING_MIDDLE}:
tokval = self.escape_brackets(tokval)
# Insert a space between two consecutive brackets if we are in an f-string

6
Makefile.pre.in
View File

@ -537,6 +537,7 @@ OBJECT_OBJS= \
Objects/floatobject.o \
Objects/frameobject.o \
Objects/funcobject.o \
Objects/interpolationobject.o \
Objects/iterobject.o \
Objects/listobject.o \
Objects/longobject.o \
@ -553,6 +554,7 @@ OBJECT_OBJS= \
Objects/setobject.o \
Objects/sliceobject.o \
Objects/structseq.o \
Objects/templateobject.o \
Objects/tupleobject.o \
Objects/typeobject.o \
Objects/typevarobject.o \
@ -1323,6 +1325,7 @@ PYTHON_HEADERS= \
$(srcdir)/Include/internal/pycore_interp_structs.h \
$(srcdir)/Include/internal/pycore_interpframe.h \
$(srcdir)/Include/internal/pycore_interpframe_structs.h \
$(srcdir)/Include/internal/pycore_interpolation.h \
$(srcdir)/Include/internal/pycore_intrinsics.h \
$(srcdir)/Include/internal/pycore_jit.h \
$(srcdir)/Include/internal/pycore_list.h \
@ -1377,6 +1380,7 @@ PYTHON_HEADERS= \
$(srcdir)/Include/internal/pycore_structseq.h \
$(srcdir)/Include/internal/pycore_symtable.h \
$(srcdir)/Include/internal/pycore_sysmodule.h \
$(srcdir)/Include/internal/pycore_template.h \
$(srcdir)/Include/internal/pycore_time.h \
$(srcdir)/Include/internal/pycore_token.h \
$(srcdir)/Include/internal/pycore_traceback.h \
@ -2525,6 +2529,7 @@ LIBSUBDIRS= asyncio \
re \
site-packages \
sqlite3 \
string \
sysconfig \
tkinter \
tomllib \
@ -2635,6 +2640,7 @@ TESTSUBDIRS= idlelib/idle_test \
test/test_peg_generator \
test/test_pydoc \
test/test_pyrepl \
test/test_string \
test/test_sqlite3 \
test/test_tkinter \
test/test_tomllib \

3
Misc/NEWS.d/next/Core_and_Builtins/2025-04-22-15-37-05.gh-issue-132661.XE_A42.rst Normal file
View File

@ -0,0 +1,3 @@
Implement :pep:`750` (Template Strings). Add new syntax for t-strings and
implement new internal :class:`!string.templatelib.Template` and
:class:`!string.templatelib.Interpolation` types.

89
Objects/clinic/interpolationobject.c.h generated Normal file
View File

@ -0,0 +1,89 @@
/*[clinic input]
preserve
[clinic start generated code]*/
#if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE)
# include "pycore_gc.h" // PyGC_Head
# include "pycore_runtime.h" // _Py_ID()
#endif
#include "pycore_modsupport.h" // _PyArg_UnpackKeywords()
static PyObject *
interpolation_new_impl(PyTypeObject *type, PyObject *value,
PyObject *expression, PyObject *conversion,
PyObject *format_spec);
static PyObject *
interpolation_new(PyTypeObject *type, PyObject *args, PyObject *kwargs)
{
PyObject *return_value = NULL;
#if defined(Py_BUILD_CORE) && !defined(Py_BUILD_CORE_MODULE)
#define NUM_KEYWORDS 4
static struct {
PyGC_Head _this_is_not_used;
PyObject_VAR_HEAD
Py_hash_t ob_hash;
PyObject *ob_item[NUM_KEYWORDS];
} _kwtuple = {
.ob_base = PyVarObject_HEAD_INIT(&PyTuple_Type, NUM_KEYWORDS)
.ob_hash = -1,
.ob_item = { &_Py_ID(value), &_Py_ID(expression), &_Py_ID(conversion), &_Py_ID(format_spec), },
};
#undef NUM_KEYWORDS
#define KWTUPLE (&_kwtuple.ob_base.ob_base)
#else // !Py_BUILD_CORE
# define KWTUPLE NULL
#endif // !Py_BUILD_CORE
static const char * const _keywords[] = {"value", "expression", "conversion", "format_spec", NULL};
static _PyArg_Parser _parser = {
.keywords = _keywords,
.fname = "Interpolation",
.kwtuple = KWTUPLE,
};
#undef KWTUPLE
PyObject *argsbuf[4];
PyObject * const *fastargs;
Py_ssize_t nargs = PyTuple_GET_SIZE(args);
Py_ssize_t noptargs = nargs + (kwargs ? PyDict_GET_SIZE(kwargs) : 0) - 2;
PyObject *value;
PyObject *expression;
PyObject *conversion = Py_None;
PyObject *format_spec = &_Py_STR(empty);
fastargs = _PyArg_UnpackKeywords(_PyTuple_CAST(args)->ob_item, nargs, kwargs, NULL, &_parser,
/*minpos*/ 2, /*maxpos*/ 4, /*minkw*/ 0, /*varpos*/ 0, argsbuf);
if (!fastargs) {
goto exit;
}
value = fastargs[0];
if (!PyUnicode_Check(fastargs[1])) {
_PyArg_BadArgument("Interpolation", "argument 'expression'", "str", fastargs[1]);
goto exit;
}
expression = fastargs[1];
if (!noptargs) {
goto skip_optional_pos;
}
if (fastargs[2]) {
if (!_conversion_converter(fastargs[2], &conversion)) {
goto exit;
}
if (!--noptargs) {
goto skip_optional_pos;
}
}
if (!PyUnicode_Check(fastargs[3])) {
_PyArg_BadArgument("Interpolation", "argument 'format_spec'", "str", fastargs[3]);
goto exit;
}
format_spec = fastargs[3];
skip_optional_pos:
return_value = interpolation_new_impl(type, value, expression, conversion, format_spec);
exit:
return return_value;
}
/*[clinic end generated code: output=599742a5ccd6f060 input=a9049054013a1b77]*/

229
Objects/interpolationobject.c Normal file
View File

@ -0,0 +1,229 @@
/* t-string Interpolation object implementation */
#include "Python.h"
#include "pycore_initconfig.h" // _PyStatus_OK
#include "pycore_interpolation.h"
#include "pycore_typeobject.h" // _PyType_GetDict
static int
_conversion_converter(PyObject *arg, PyObject **conversion)
{
if (arg == Py_None) {
return 1;
}
if (!PyUnicode_Check(arg)) {
PyErr_Format(PyExc_TypeError,
"Interpolation() argument 'conversion' must be str, not %T",
arg);
return 0;
}
Py_ssize_t len;
const char *conv_str = PyUnicode_AsUTF8AndSize(arg, &len);
if (len != 1 || !(conv_str[0] == 'a' || conv_str[0] == 'r' || conv_str[0] == 's')) {
PyErr_SetString(PyExc_ValueError,
"Interpolation() argument 'conversion' must be one of 's', 'a' or 'r'");
return 0;
}
*conversion = arg;
return 1;
}
#include "clinic/interpolationobject.c.h"
/*[clinic input]
class Interpolation "interpolationobject *" "&_PyInterpolation_Type"
[clinic start generated code]*/
/*[clinic end generated code: output=da39a3ee5e6b4b0d input=161c64a16f9c4544]*/
typedef struct {
PyObject_HEAD
PyObject *value;
PyObject *expression;
PyObject *conversion;
PyObject *format_spec;
} interpolationobject;
#define interpolationobject_CAST(op) \
(assert(_PyInterpolation_CheckExact(op)), _Py_CAST(interpolationobject*, (op)))
/*[clinic input]
@classmethod
Interpolation.__new__ as interpolation_new
value: object
expression: object(subclass_of='&PyUnicode_Type')
conversion: object(converter='_conversion_converter') = None
format_spec: object(subclass_of='&PyUnicode_Type', c_default='&_Py_STR(empty)') = ""
[clinic start generated code]*/
static PyObject *
interpolation_new_impl(PyTypeObject *type, PyObject *value,
PyObject *expression, PyObject *conversion,
PyObject *format_spec)
/*[clinic end generated code: output=6488e288765bc1a9 input=d91711024068528c]*/
{
interpolationobject *self = PyObject_GC_New(interpolationobject, type);
if (!self) {
return NULL;
}
self->value = Py_NewRef(value);
self->expression = Py_NewRef(expression);
self->conversion = Py_NewRef(conversion);
self->format_spec = Py_NewRef(format_spec);
PyObject_GC_Track(self);
return (PyObject *) self;
}
static void
interpolation_dealloc(PyObject *op)
{
PyObject_GC_UnTrack(op);
Py_TYPE(op)->tp_clear(op);
Py_TYPE(op)->tp_free(op);
}
static int
interpolation_clear(PyObject *op)
{
interpolationobject *self = interpolationobject_CAST(op);
Py_CLEAR(self->value);
Py_CLEAR(self->expression);
Py_CLEAR(self->conversion);
Py_CLEAR(self->format_spec);
return 0;
}
static int
interpolation_traverse(PyObject *op, visitproc visit, void *arg)
{
interpolationobject *self = interpolationobject_CAST(op);
Py_VISIT(self->value);
Py_VISIT(self->expression);
Py_VISIT(self->conversion);
Py_VISIT(self->format_spec);
return 0;
}
static PyObject *
interpolation_repr(PyObject *op)
{
interpolationobject *self = interpolationobject_CAST(op);
return PyUnicode_FromFormat("%s(%R, %R, %R, %R)",
_PyType_Name(Py_TYPE(self)), self->value, self->expression,
self->conversion, self->format_spec);
}
static PyMemberDef interpolation_members[] = {
{"value", Py_T_OBJECT_EX, offsetof(interpolationobject, value), Py_READONLY, "Value"},
{"expression", Py_T_OBJECT_EX, offsetof(interpolationobject, expression), Py_READONLY, "Expression"},
{"conversion", Py_T_OBJECT_EX, offsetof(interpolationobject, conversion), Py_READONLY, "Conversion"},
{"format_spec", Py_T_OBJECT_EX, offsetof(interpolationobject, format_spec), Py_READONLY, "Format specifier"},
{NULL}
};
static PyObject*
interpolation_reduce(PyObject *op, PyObject *Py_UNUSED(dummy))
{
interpolationobject *self = interpolationobject_CAST(op);
return Py_BuildValue("(O(OOOO))", (PyObject *)Py_TYPE(op),
self->value, self->expression,
self->conversion, self->format_spec);
}
static PyMethodDef interpolation_methods[] = {
{"__reduce__", interpolation_reduce, METH_NOARGS,
PyDoc_STR("__reduce__() -> (cls, state)")},
{NULL, NULL},
};
PyTypeObject _PyInterpolation_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
.tp_name = "string.templatelib.Interpolation",
.tp_doc = PyDoc_STR("Interpolation object"),
.tp_basicsize = sizeof(interpolationobject),
.tp_itemsize = 0,
.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
.tp_new = interpolation_new,
.tp_alloc = PyType_GenericAlloc,
.tp_dealloc = interpolation_dealloc,
.tp_clear = interpolation_clear,
.tp_free = PyObject_GC_Del,
.tp_repr = interpolation_repr,
.tp_members = interpolation_members,
.tp_methods = interpolation_methods,
.tp_traverse = interpolation_traverse,
};
PyStatus
_PyInterpolation_InitTypes(PyInterpreterState *interp)
{
PyObject *tuple = Py_BuildValue("(ssss)", "value", "expression", "conversion", "format_spec");
if (!tuple) {
goto error;
}
PyObject *dict = _PyType_GetDict(&_PyInterpolation_Type);
if (!dict) {
Py_DECREF(tuple);
goto error;
}
int status = PyDict_SetItemString(dict, "__match_args__", tuple);
Py_DECREF(tuple);
if (status < 0) {
goto error;
}
return _PyStatus_OK();
error:
return _PyStatus_ERR("Can't initialize interpolation types");
}
PyObject *
_PyInterpolation_Build(PyObject *value, PyObject *str, int conversion, PyObject *format_spec)
{
interpolationobject *interpolation = PyObject_GC_New(interpolationobject, &_PyInterpolation_Type);
if (!interpolation) {
return NULL;
}
interpolation->value = Py_NewRef(value);
interpolation->expression = Py_NewRef(str);
interpolation->format_spec = Py_NewRef(format_spec);
interpolation->conversion = NULL;
if (conversion == 0) {
interpolation->conversion = Py_None;
}
else {
switch (conversion) {
case FVC_ASCII:
interpolation->conversion = _Py_LATIN1_CHR('a');
break;
case FVC_REPR:
interpolation->conversion = _Py_LATIN1_CHR('r');
break;
case FVC_STR:
interpolation->conversion = _Py_LATIN1_CHR('s');
break;
default:
PyErr_SetString(PyExc_SystemError,
"Interpolation() argument 'conversion' must be one of 's', 'a' or 'r'");
Py_DECREF(interpolation);
return NULL;
}
}
PyObject_GC_Track(interpolation);
return (PyObject *) interpolation;
}
PyObject *
_PyInterpolation_GetValueRef(PyObject *interpolation)
{
return Py_NewRef(interpolationobject_CAST(interpolation)->value);
}

5
Objects/object.c
View File

@ -15,6 +15,7 @@
#include "pycore_hamt.h" // _PyHamtItems_Type
#include "pycore_initconfig.h" // _PyStatus_OK()
#include "pycore_instruction_sequence.h" // _PyInstructionSequence_Type
#include "pycore_interpolation.h" // _PyInterpolation_Type
#include "pycore_list.h" // _PyList_DebugMallocStats()
#include "pycore_long.h" // _PyLong_GetZero()
#include "pycore_memoryobject.h" // _PyManagedBuffer_Type
@ -25,6 +26,7 @@
#include "pycore_pymem.h" // _PyMem_IsPtrFreed()
#include "pycore_pystate.h" // _PyThreadState_GET()
#include "pycore_symtable.h" // PySTEntry_Type
#include "pycore_template.h" // _PyTemplate_Type _PyTemplateIter_Type
#include "pycore_tuple.h" // _PyTuple_DebugMallocStats()
#include "pycore_typeobject.h" // _PyBufferWrapper_Type
#include "pycore_typevarobject.h" // _PyTypeAlias_Type
@ -2409,6 +2411,7 @@ static PyTypeObject* static_types[] = {
&_PyHamt_CollisionNode_Type,
&_PyHamt_Type,
&_PyInstructionSequence_Type,
&_PyInterpolation_Type,
&_PyLegacyEventHandler_Type,
&_PyLineIterator,
&_PyManagedBuffer_Type,
@ -2418,6 +2421,8 @@ static PyTypeObject* static_types[] = {
&_PyNone_Type,
&_PyNotImplemented_Type,
&_PyPositionsIterator,
&_PyTemplate_Type,
&_PyTemplateIter_Type,
&_PyUnicodeASCIIIter_Type,
&_PyUnion_Type,
#ifdef _Py_TIER2

483
Objects/templateobject.c Normal file
View File

@ -0,0 +1,483 @@
/* t-string Template object implementation */
#include "Python.h"
#include "pycore_interpolation.h" // _PyInterpolation_CheckExact()
#include "pycore_runtime.h" // _Py_STR()
#include "pycore_template.h"
typedef struct {
PyObject_HEAD
PyObject *stringsiter;
PyObject *interpolationsiter;
int from_strings;
} templateiterobject;
#define templateiterobject_CAST(op) \
(assert(_PyTemplateIter_CheckExact(op)), _Py_CAST(templateiterobject*, (op)))
static PyObject *
templateiter_next(PyObject *op)
{
templateiterobject *self = templateiterobject_CAST(op);
PyObject *item;
if (self->from_strings) {
item = PyIter_Next(self->stringsiter);
self->from_strings = 0;
if (PyUnicode_GET_LENGTH(item) == 0) {
Py_SETREF(item, PyIter_Next(self->interpolationsiter));
self->from_strings = 1;
}
} else {
item = PyIter_Next(self->interpolationsiter);
self->from_strings = 1;
}
return item;
}
static void
templateiter_dealloc(PyObject *op)
{
PyObject_GC_UnTrack(op);
Py_TYPE(op)->tp_clear(op);
Py_TYPE(op)->tp_free(op);
}
static int
templateiter_clear(PyObject *op)
{
templateiterobject *self = templateiterobject_CAST(op);
Py_CLEAR(self->stringsiter);
Py_CLEAR(self->interpolationsiter);
return 0;
}
static int
templateiter_traverse(PyObject *op, visitproc visit, void *arg)
{
templateiterobject *self = templateiterobject_CAST(op);
Py_VISIT(self->stringsiter);
Py_VISIT(self->interpolationsiter);
return 0;
}
PyTypeObject _PyTemplateIter_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
.tp_name = "string.templatelib.TemplateIter",
.tp_doc = PyDoc_STR("Template iterator object"),
.tp_basicsize = sizeof(templateiterobject),
.tp_itemsize = 0,
.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
.tp_alloc = PyType_GenericAlloc,
.tp_dealloc = templateiter_dealloc,
.tp_clear = templateiter_clear,
.tp_free = PyObject_GC_Del,
.tp_traverse = templateiter_traverse,
.tp_iter = PyObject_SelfIter,
.tp_iternext = templateiter_next,
};
typedef struct {
PyObject_HEAD
PyObject *strings;
PyObject *interpolations;
} templateobject;
#define templateobject_CAST(op) \
(assert(_PyTemplate_CheckExact(op)), _Py_CAST(templateobject*, (op)))
static PyObject *
template_new(PyTypeObject *type, PyObject *args, PyObject *kwds)
{
if (kwds != NULL) {
PyErr_SetString(PyExc_TypeError, "Template.__new__ only accepts *args arguments");
return NULL;
}
Py_ssize_t argslen = PyTuple_GET_SIZE(args);
Py_ssize_t stringslen = 0;
Py_ssize_t interpolationslen = 0;
int last_was_str = 0;
for (Py_ssize_t i = 0; i < argslen; i++) {
PyObject *item = PyTuple_GET_ITEM(args, i);
if (PyUnicode_Check(item)) {
if (!last_was_str) {
stringslen++;
}
last_was_str = 1;
}
else if (_PyInterpolation_CheckExact(item)) {
if (!last_was_str) {
stringslen++;
}
interpolationslen++;
last_was_str = 0;
}
else {
PyErr_Format(
PyExc_TypeError,
"Template.__new__ *args need to be of type 'str' or 'Interpolation', got %T",
item);
return NULL;
}
}
if (!last_was_str) {
stringslen++;
}
PyObject *strings = PyTuple_New(stringslen);
if (!strings) {
return NULL;
}
PyObject *interpolations = PyTuple_New(interpolationslen);
if (!interpolations) {
Py_DECREF(strings);
return NULL;
}
last_was_str = 0;
Py_ssize_t stringsidx = 0, interpolationsidx = 0;
for (Py_ssize_t i = 0; i < argslen; i++) {
PyObject *item = PyTuple_GET_ITEM(args, i);
if (PyUnicode_Check(item)) {
if (last_was_str) {
PyObject *laststring = PyTuple_GET_ITEM(strings, stringsidx - 1);
PyObject *concat = PyUnicode_Concat(laststring, item);
Py_DECREF(laststring);
if (!concat) {
Py_DECREF(strings);
Py_DECREF(interpolations);
return NULL;
}
PyTuple_SET_ITEM(strings, stringsidx - 1, concat);
}
else {
PyTuple_SET_ITEM(strings, stringsidx++, Py_NewRef(item));
}
last_was_str = 1;
}
else if (_PyInterpolation_CheckExact(item)) {
if (!last_was_str) {
_Py_DECLARE_STR(empty, "");
PyTuple_SET_ITEM(strings, stringsidx++, &_Py_STR(empty));
}
PyTuple_SET_ITEM(interpolations, interpolationsidx++, Py_NewRef(item));
last_was_str = 0;
}
}
if (!last_was_str) {
_Py_DECLARE_STR(empty, "");
PyTuple_SET_ITEM(strings, stringsidx++, &_Py_STR(empty));
}
PyObject *template = _PyTemplate_Build(strings, interpolations);
Py_DECREF(strings);
Py_DECREF(interpolations);
return template;
}
static void
template_dealloc(PyObject *op)
{
PyObject_GC_UnTrack(op);
Py_TYPE(op)->tp_clear(op);
Py_TYPE(op)->tp_free(op);
}
static int
template_clear(PyObject *op)
{
templateobject *self = templateobject_CAST(op);
Py_CLEAR(self->strings);
Py_CLEAR(self->interpolations);
return 0;
}
static int
template_traverse(PyObject *op, visitproc visit, void *arg)
{
templateobject *self = templateobject_CAST(op);
Py_VISIT(self->strings);
Py_VISIT(self->interpolations);
return 0;
}
static PyObject *
template_repr(PyObject *op)
{
templateobject *self = templateobject_CAST(op);
return PyUnicode_FromFormat("%s(strings=%R, interpolations=%R)",
_PyType_Name(Py_TYPE(self)),
self->strings,
self->interpolations);
}
static PyObject *
template_iter(PyObject *op)
{
templateobject *self = templateobject_CAST(op);
templateiterobject *iter = PyObject_GC_New(templateiterobject, &_PyTemplateIter_Type);
if (iter == NULL) {
return NULL;
}
PyObject *stringsiter = PyObject_GetIter(self->strings);
if (stringsiter == NULL) {
Py_DECREF(iter);
return NULL;
}
PyObject *interpolationsiter = PyObject_GetIter(self->interpolations);
if (interpolationsiter == NULL) {
Py_DECREF(iter);
Py_DECREF(stringsiter);
return NULL;
}
iter->stringsiter = stringsiter;
iter->interpolationsiter = interpolationsiter;
iter->from_strings = 1;
PyObject_GC_Track(iter);
return (PyObject *)iter;
}
static PyObject *
template_strings_append_str(PyObject *strings, PyObject *str)
{
Py_ssize_t stringslen = PyTuple_GET_SIZE(strings);
PyObject *string = PyTuple_GET_ITEM(strings, stringslen - 1);
PyObject *concat = PyUnicode_Concat(string, str);
if (concat == NULL) {
return NULL;
}
PyObject *newstrings = PyTuple_New(stringslen);
if (newstrings == NULL) {
Py_DECREF(concat);
return NULL;
}
for (Py_ssize_t i = 0; i < stringslen - 1; i++) {
PyTuple_SET_ITEM(newstrings, i, Py_NewRef(PyTuple_GET_ITEM(strings, i)));
}
PyTuple_SET_ITEM(newstrings, stringslen - 1, concat);
return newstrings;
}
static PyObject *
template_strings_prepend_str(PyObject *strings, PyObject *str)
{
Py_ssize_t stringslen = PyTuple_GET_SIZE(strings);
PyObject *string = PyTuple_GET_ITEM(strings, 0);
PyObject *concat = PyUnicode_Concat(str, string);
if (concat == NULL) {
return NULL;
}
PyObject *newstrings = PyTuple_New(stringslen);
if (newstrings == NULL) {
Py_DECREF(concat);
return NULL;
}
PyTuple_SET_ITEM(newstrings, 0, concat);
for (Py_ssize_t i = 1; i < stringslen; i++) {
PyTuple_SET_ITEM(newstrings, i, Py_NewRef(PyTuple_GET_ITEM(strings, i)));
}
return newstrings;
}
static PyObject *
template_strings_concat(PyObject *left, PyObject *right)
{
Py_ssize_t left_stringslen = PyTuple_GET_SIZE(left);
PyObject *left_laststring = PyTuple_GET_ITEM(left, left_stringslen - 1);
Py_ssize_t right_stringslen = PyTuple_GET_SIZE(right);
PyObject *right_firststring = PyTuple_GET_ITEM(right, 0);
PyObject *concat = PyUnicode_Concat(left_laststring, right_firststring);
if (concat == NULL) {
return NULL;
}
PyObject *newstrings = PyTuple_New(left_stringslen + right_stringslen - 1);
if (newstrings == NULL) {
Py_DECREF(concat);
return NULL;
}
Py_ssize_t index = 0;
for (Py_ssize_t i = 0; i < left_stringslen - 1; i++) {
PyTuple_SET_ITEM(newstrings, index++, Py_NewRef(PyTuple_GET_ITEM(left, i)));
}
PyTuple_SET_ITEM(newstrings, index++, concat);
for (Py_ssize_t i = 1; i < right_stringslen; i++) {
PyTuple_SET_ITEM(newstrings, index++, Py_NewRef(PyTuple_GET_ITEM(right, i)));
}
return newstrings;
}
static PyObject *
template_concat_templates(templateobject *self, templateobject *other)
{
PyObject *newstrings = template_strings_concat(self->strings, other->strings);
if (newstrings == NULL) {
return NULL;
}
PyObject *newinterpolations = PySequence_Concat(self->interpolations, other->interpolations);
if (newinterpolations == NULL) {
Py_DECREF(newstrings);
return NULL;
}
PyObject *newtemplate = _PyTemplate_Build(newstrings, newinterpolations);
Py_DECREF(newstrings);
Py_DECREF(newinterpolations);
return newtemplate;
}
static PyObject *
template_concat_template_str(templateobject *self, PyObject *other)
{
PyObject *newstrings = template_strings_append_str(self->strings, other);
if (newstrings == NULL) {
return NULL;
}
PyObject *newtemplate = _PyTemplate_Build(newstrings, self->interpolations);
Py_DECREF(newstrings);
return newtemplate;
}
static PyObject *
template_concat_str_template(templateobject *self, PyObject *other)
{
PyObject *newstrings = template_strings_prepend_str(self->strings, other);
if (newstrings == NULL) {
return NULL;
}
PyObject *newtemplate = _PyTemplate_Build(newstrings, self->interpolations);
Py_DECREF(newstrings);
return newtemplate;
}
PyObject *
_PyTemplate_Concat(PyObject *self, PyObject *other)
{
if (_PyTemplate_CheckExact(self) && _PyTemplate_CheckExact(other)) {
return template_concat_templates((templateobject *) self, (templateobject *) other);
}
else if ((_PyTemplate_CheckExact(self)) && PyUnicode_Check(other)) {
return template_concat_template_str((templateobject *) self, other);
}
else if (PyUnicode_Check(self) && (_PyTemplate_CheckExact(other))) {
return template_concat_str_template((templateobject *) other, self);
}
else {
Py_RETURN_NOTIMPLEMENTED;
}
}
static PyObject *
template_values_get(PyObject *op, void *Py_UNUSED(data))
{
templateobject *self = templateobject_CAST(op);
Py_ssize_t len = PyTuple_GET_SIZE(self->interpolations);
PyObject *values = PyTuple_New(PyTuple_GET_SIZE(self->interpolations));
if (values == NULL) {
return NULL;
}
for (Py_ssize_t i = 0; i < len; i++) {
PyObject *item = PyTuple_GET_ITEM(self->interpolations, i);
PyTuple_SET_ITEM(values, i, _PyInterpolation_GetValueRef(item));
}
return values;
}
static PyMemberDef template_members[] = {
{"strings", Py_T_OBJECT_EX, offsetof(templateobject, strings), Py_READONLY, "Strings"},
{"interpolations", Py_T_OBJECT_EX, offsetof(templateobject, interpolations), Py_READONLY, "Interpolations"},
{NULL},
};
static PyGetSetDef template_getset[] = {
{"values", template_values_get, NULL,
PyDoc_STR("Values of interpolations"), NULL},
{NULL},
};
static PySequenceMethods template_as_sequence = {
.sq_concat = _PyTemplate_Concat,
};
static PyObject*
template_reduce(PyObject *op, PyObject *Py_UNUSED(dummy))
{
PyObject *mod = PyImport_ImportModule("string.templatelib");
if (mod == NULL) {
return NULL;
}
PyObject *func = PyObject_GetAttrString(mod, "_template_unpickle");
Py_DECREF(mod);
if (func == NULL) {
return NULL;
}
templateobject *self = templateobject_CAST(op);
PyObject *result = Py_BuildValue("O(OO)",
func,
self->strings,
self->interpolations);
Py_DECREF(func);
return result;
}
static PyMethodDef template_methods[] = {
{"__reduce__", template_reduce, METH_NOARGS, NULL},
{NULL, NULL},
};
PyTypeObject _PyTemplate_Type = {
PyVarObject_HEAD_INIT(NULL, 0)
.tp_name = "string.templatelib.Template",
.tp_doc = PyDoc_STR("Template object"),
.tp_basicsize = sizeof(templateobject),
.tp_itemsize = 0,
.tp_flags = Py_TPFLAGS_DEFAULT | Py_TPFLAGS_HAVE_GC,
.tp_as_sequence = &template_as_sequence,
.tp_new = template_new,
.tp_alloc = PyType_GenericAlloc,
.tp_dealloc = template_dealloc,
.tp_clear = template_clear,
.tp_free = PyObject_GC_Del,
.tp_repr = template_repr,
.tp_members = template_members,
.tp_methods = template_methods,
.tp_getset = template_getset,
.tp_iter = template_iter,
.tp_traverse = template_traverse,
};
PyObject *
_PyTemplate_Build(PyObject *strings, PyObject *interpolations)
{
templateobject *template = PyObject_GC_New(templateobject, &_PyTemplate_Type);
if (template == NULL) {
return NULL;
}
template->strings = Py_NewRef(strings);
template->interpolations = Py_NewRef(interpolations);
PyObject_GC_Track(template);
return (PyObject *) template;
}

15
Objects/unicodeobject.c
View File

@ -56,6 +56,7 @@ OF OR IN CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
#include "pycore_pyhash.h" // _Py_HashSecret_t
#include "pycore_pylifecycle.h" // _Py_SetFileSystemEncoding()
#include "pycore_pystate.h" // _PyInterpreterState_GET()
#include "pycore_template.h" // _PyTemplate_Concat()
#include "pycore_tuple.h" // _PyTuple_FromArray()
#include "pycore_ucnhash.h" // _PyUnicode_Name_CAPI
#include "pycore_unicodeobject.h" // struct _Py_unicode_state
@ -11628,10 +11629,16 @@ PyUnicode_Concat(PyObject *left, PyObject *right)
return NULL;
if (!PyUnicode_Check(right)) {
PyErr_Format(PyExc_TypeError,
"can only concatenate str (not \"%.200s\") to str",
Py_TYPE(right)->tp_name);
return NULL;
if (_PyTemplate_CheckExact(right)) {
// str + tstring is implemented in the tstring type
return _PyTemplate_Concat(left, right);
}
else {
PyErr_Format(PyExc_TypeError,
"can only concatenate str (not \"%.200s\") to str",
Py_TYPE(right)->tp_name);
return NULL;
}
}
/* Shortcuts */

2
PCbuild/_freeze_module.vcxproj
View File

@ -143,6 +143,7 @@
<ClCompile Include="..\Objects\funcobject.c" />
<ClCompile Include="..\Objects\genericaliasobject.c" />
<ClCompile Include="..\Objects\genobject.c" />
<ClCompile Include="..\Objects\interpolationobject.c" />
<ClCompile Include="..\Objects\iterobject.c" />
<ClCompile Include="..\Objects\listobject.c" />
<ClCompile Include="..\Objects\longobject.c" />
@ -158,6 +159,7 @@
<ClCompile Include="..\Objects\setobject.c" />
<ClCompile Include="..\Objects\sliceobject.c" />
<ClCompile Include="..\Objects\structseq.c" />
<ClCompile Include="..\Objects\templateobject.c" />
<ClCompile Include="..\Objects\tupleobject.c" />
<ClCompile Include="..\Objects\typeobject.c" />
<ClCompile Include="..\Objects\typevarobject.c" />

4
PCbuild/pythoncore.vcxproj
View File

@ -264,6 +264,7 @@
<ClInclude Include="..\Include\internal\pycore_interp_structs.h" />
<ClInclude Include="..\Include\internal\pycore_interpframe.h" />
<ClInclude Include="..\Include\internal\pycore_interpframe_structs.h" />
<ClInclude Include="..\Include\internal\pycore_interpolation.h" />
<ClInclude Include="..\Include\internal\pycore_intrinsics.h" />
<ClInclude Include="..\Include\internal\pycore_jit.h" />
<ClInclude Include="..\Include\internal\pycore_list.h" />
@ -310,6 +311,7 @@
<ClInclude Include="..\Include\internal\pycore_structseq.h" />
<ClInclude Include="..\Include\internal\pycore_sysmodule.h" />
<ClInclude Include="..\Include\internal\pycore_symtable.h" />
<ClInclude Include="..\Include\internal\pycore_template.h" />
<ClInclude Include="..\Include\internal\pycore_time.h" />
<ClInclude Include="..\Include\internal\pycore_token.h" />
<ClInclude Include="..\Include\internal\pycore_traceback.h" />
@ -528,6 +530,7 @@
<ClCompile Include="..\Objects\funcobject.c" />
<ClCompile Include="..\Objects\genericaliasobject.c" />
<ClCompile Include="..\Objects\genobject.c" />
<ClCompile Include="..\Objects\interpolationobject.c" />
<ClCompile Include="..\Objects\iterobject.c" />
<ClCompile Include="..\Objects\listobject.c" />
<ClCompile Include="..\Objects\longobject.c" />
@ -543,6 +546,7 @@
<ClCompile Include="..\Objects\setobject.c" />
<ClCompile Include="..\Objects\sliceobject.c" />
<ClCompile Include="..\Objects\structseq.c" />
<ClCompile Include="..\Objects\templateobject.c" />
<ClCompile Include="..\Objects\tupleobject.c" />
<ClCompile Include="..\Objects\typeobject.c" />
<ClCompile Include="..\Objects\typevarobject.c" />

6
PCbuild/pythoncore.vcxproj.filters
View File

@ -708,6 +708,9 @@
<ClInclude Include="..\Include\internal\pycore_interpframe_structs.h">
<Filter>Include\internal</Filter>
</ClInclude>
<ClInclude Include="..\Include\internal\pycore_interpolation.h">
<Filter>Include\internal</Filter>
</ClInclude>
<ClInclude Include="..\Include\internal\pycore_intrinsics.h">
<Filter>Include\cpython</Filter>
</ClInclude>
@ -840,6 +843,9 @@
<ClInclude Include="..\Include\internal\pycore_stackref.h">
<Filter>Include\internal</Filter>
</ClInclude>
<ClInclude Include="..\Include\internal\pycore_template.h">
<Filter>Include\internal</Filter>
</ClInclude>
<ClInclude Include="..\Include\internal\pycore_time.h">
<Filter>Include\internal</Filter>
</ClInclude>

2
Parser/Python.asdl
View File

@ -78,7 +78,9 @@ module Python
| Compare(expr left, cmpop* ops, expr* comparators)
| Call(expr func, expr* args, keyword* keywords)
| FormattedValue(expr value, int conversion, expr? format_spec)
| Interpolation(expr value, constant str, int conversion, expr? format_spec)
| JoinedStr(expr* values)
| TemplateStr(expr* values)
| Constant(constant value, string? kind)
-- the following expression can appear in assignment context

470
Parser/action_helpers.c
View File

@ -965,9 +965,21 @@ _PyPegen_check_fstring_conversion(Parser *p, Token* conv_token, expr_ty conv)
if (conv_token->lineno != conv->lineno || conv_token->end_col_offset != conv->col_offset) {
return RAISE_SYNTAX_ERROR_KNOWN_RANGE(
conv_token, conv,
"f-string: conversion type must come right after the exclamanation mark"
"%c-string: conversion type must come right after the exclamanation mark",
TOK_GET_STRING_PREFIX(p->tok)
);
}
Py_UCS4 first = PyUnicode_READ_CHAR(conv->v.Name.id, 0);
if (PyUnicode_GET_LENGTH(conv->v.Name.id) > 1 ||
!(first == 's' || first == 'r' || first == 'a')) {
RAISE_SYNTAX_ERROR_KNOWN_LOCATION(conv,
"%c-string: invalid conversion character %R: expected 's', 'r', or 'a'",
TOK_GET_STRING_PREFIX(p->tok),
conv->v.Name.id);
return NULL;
}
return result_token_with_metadata(p, conv, conv_token->metadata);
}
@ -1070,6 +1082,9 @@ _PyPegen_get_expr_name(expr_ty e)
case JoinedStr_kind:
case FormattedValue_kind:
return "f-string expression";
case TemplateStr_kind:
case Interpolation_kind:
return "t-string expression";
case Constant_kind: {
PyObject *value = e->v.Constant.value;
if (value == Py_None) {
@ -1279,20 +1294,13 @@ _PyPegen_decode_fstring_part(Parser* p, int is_raw, expr_ty constant, Token* tok
p->arena);
}
expr_ty
_PyPegen_joined_str(Parser *p, Token* a, asdl_expr_seq* expr, Token*b) {
/* The parser might put multiple f-string values into an individual
* JoinedStr node at the top level due to stuff like f-string debugging
* expressions. This function flattens those and promotes them to the
* upper level. Only simplifies AST, but the compiler already takes care
* of the regular output, so this is not necessary if you are not going
* to expose the output AST to Python level. */
Py_ssize_t n_items = asdl_seq_LEN(expr);
static asdl_expr_seq *
_get_resized_exprs(Parser *p, Token *a, asdl_expr_seq *raw_expressions, Token *b, enum string_kind_t string_kind)
{
Py_ssize_t n_items = asdl_seq_LEN(raw_expressions);
Py_ssize_t total_items = n_items;
for (Py_ssize_t i = 0; i < n_items; i++) {
expr_ty item = asdl_seq_GET(expr, i);
expr_ty item = asdl_seq_GET(raw_expressions, i);
if (item->kind == JoinedStr_kind) {
total_items += asdl_seq_LEN(item->v.JoinedStr.values) - 1;
}
@ -1311,17 +1319,19 @@ _PyPegen_joined_str(Parser *p, Token* a, asdl_expr_seq* expr, Token*b) {
Py_ssize_t index = 0;
for (Py_ssize_t i = 0; i < n_items; i++) {
expr_ty item = asdl_seq_GET(expr, i);
expr_ty item = asdl_seq_GET(raw_expressions, i);
// This should correspond to a JoinedStr node of two elements
// created _PyPegen_formatted_value. This situation can only be the result of
// a f-string debug expression where the first element is a constant with the text and the second
// a (f|t)-string debug expression where the first element is a constant with the text and the second
// a formatted value with the expression.
if (item->kind == JoinedStr_kind) {
asdl_expr_seq *values = item->v.JoinedStr.values;
if (asdl_seq_LEN(values) != 2) {
PyErr_Format(PyExc_SystemError,
"unexpected JoinedStr node without debug data in f-string at line %d",
string_kind == TSTRING
? "unexpected TemplateStr node without debug data in t-string at line %d"
: "unexpected JoinedStr node without debug data in f-string at line %d",
item->lineno);
return NULL;
}
@ -1331,7 +1341,7 @@ _PyPegen_joined_str(Parser *p, Token* a, asdl_expr_seq* expr, Token*b) {
asdl_seq_SET(seq, index++, first);
expr_ty second = asdl_seq_GET(values, 1);
assert(second->kind == FormattedValue_kind);
assert((string_kind == TSTRING && second->kind == Interpolation_kind) || second->kind == FormattedValue_kind);
asdl_seq_SET(seq, index++, second);
continue;
@ -1367,7 +1377,22 @@ _PyPegen_joined_str(Parser *p, Token* a, asdl_expr_seq* expr, Token*b) {
else {
resized_exprs = seq;
}
return resized_exprs;
}
expr_ty
_PyPegen_template_str(Parser *p, Token *a, asdl_expr_seq *raw_expressions, Token *b) {
asdl_expr_seq *resized_exprs = _get_resized_exprs(p, a, raw_expressions, b, TSTRING);
return _PyAST_TemplateStr(resized_exprs, a->lineno, a->col_offset,
b->end_lineno, b->end_col_offset,
p->arena);
}
expr_ty
_PyPegen_joined_str(Parser *p, Token* a, asdl_expr_seq* raw_expressions, Token*b) {
asdl_expr_seq *resized_exprs = _get_resized_exprs(p, a, raw_expressions, b, FSTRING);
return _PyAST_JoinedStr(resized_exprs, a->lineno, a->col_offset,
b->end_lineno, b->end_col_offset,
p->arena);
@ -1434,29 +1459,101 @@ expr_ty _PyPegen_constant_from_string(Parser* p, Token* tok) {
return _PyAST_Constant(s, kind, tok->lineno, tok->col_offset, tok->end_lineno, tok->end_col_offset, p->arena);
}
expr_ty _PyPegen_formatted_value(Parser *p, expr_ty expression, Token *debug, ResultTokenWithMetadata *conversion,
ResultTokenWithMetadata *format, Token *closing_brace, int lineno, int col_offset,
int end_lineno, int end_col_offset, PyArena *arena) {
int conversion_val = -1;
static int
_get_interpolation_conversion(Parser *p, Token *debug, ResultTokenWithMetadata *conversion,
ResultTokenWithMetadata *format)
{
if (conversion != NULL) {
expr_ty conversion_expr = (expr_ty) conversion->result;
assert(conversion_expr->kind == Name_kind);
Py_UCS4 first = PyUnicode_READ_CHAR(conversion_expr->v.Name.id, 0);
if (PyUnicode_GET_LENGTH(conversion_expr->v.Name.id) > 1 ||
!(first == 's' || first == 'r' || first == 'a')) {
RAISE_SYNTAX_ERROR_KNOWN_LOCATION(conversion_expr,
"f-string: invalid conversion character %R: expected 's', 'r', or 'a'",
conversion_expr->v.Name.id);
return NULL;
}
conversion_val = Py_SAFE_DOWNCAST(first, Py_UCS4, int);
return Py_SAFE_DOWNCAST(first, Py_UCS4, int);
}
else if (debug && !format) {
/* If no conversion is specified, use !r for debug expressions */
conversion_val = (int)'r';
return (int)'r';
}
return -1;
}
static PyObject *
_strip_interpolation_expr(PyObject *exprstr)
{
Py_ssize_t len = PyUnicode_GET_LENGTH(exprstr);
for (Py_ssize_t i = len - 1; i >= 0; i--) {
Py_UCS4 c = PyUnicode_READ_CHAR(exprstr, i);
if (_PyUnicode_IsWhitespace(c) || c == '=') {
len--;
}
else {
break;
}
}
return PyUnicode_Substring(exprstr, 0, len);
}
expr_ty _PyPegen_interpolation(Parser *p, expr_ty expression, Token *debug, ResultTokenWithMetadata *conversion,
ResultTokenWithMetadata *format, Token *closing_brace, int lineno, int col_offset,
int end_lineno, int end_col_offset, PyArena *arena) {
int conversion_val = _get_interpolation_conversion(p, debug, conversion, format);
/* Find the non whitespace token after the "=" */
int debug_end_line, debug_end_offset;
PyObject *debug_metadata;
constant exprstr;
if (conversion) {
debug_end_line = ((expr_ty) conversion->result)->lineno;
debug_end_offset = ((expr_ty) conversion->result)->col_offset;
debug_metadata = exprstr = conversion->metadata;
}
else if (format) {
debug_end_line = ((expr_ty) format->result)->lineno;
debug_end_offset = ((expr_ty) format->result)->col_offset + 1;
debug_metadata = exprstr = format->metadata;
}
else {
debug_end_line = end_lineno;
debug_end_offset = end_col_offset;
debug_metadata = exprstr = closing_brace->metadata;
}
assert(exprstr != NULL);
PyObject *final_exprstr = _strip_interpolation_expr(exprstr);
if (!final_exprstr || _PyArena_AddPyObject(arena, final_exprstr) < 0) {
Py_XDECREF(final_exprstr);
return NULL;
}
expr_ty interpolation = _PyAST_Interpolation(
expression, final_exprstr, conversion_val, format ? (expr_ty) format->result : NULL,
lineno, col_offset, end_lineno,
end_col_offset, arena
);
if (!debug) {
return interpolation;
}
expr_ty debug_text = _PyAST_Constant(debug_metadata, NULL, lineno, col_offset + 1, debug_end_line,
debug_end_offset - 1, p->arena);
if (!debug_text) {
return NULL;
}
asdl_expr_seq *values = _Py_asdl_expr_seq_new(2, arena);
asdl_seq_SET(values, 0, debug_text);
asdl_seq_SET(values, 1, interpolation);
return _PyAST_JoinedStr(values, lineno, col_offset, debug_end_line, debug_end_offset, p->arena);
}
expr_ty _PyPegen_formatted_value(Parser *p, expr_ty expression, Token *debug, ResultTokenWithMetadata *conversion,
ResultTokenWithMetadata *format, Token *closing_brace, int lineno, int col_offset,
int end_lineno, int end_col_offset, PyArena *arena) {
int conversion_val = _get_interpolation_conversion(p, debug, conversion, format);
expr_ty formatted_value = _PyAST_FormattedValue(
expression, conversion_val, format ? (expr_ty) format->result : NULL,
@ -1464,108 +1561,137 @@ expr_ty _PyPegen_formatted_value(Parser *p, expr_ty expression, Token *debug, Re
end_col_offset, arena
);
if (debug) {
/* Find the non whitespace token after the "=" */
int debug_end_line, debug_end_offset;
PyObject *debug_metadata;
if (conversion) {
debug_end_line = ((expr_ty) conversion->result)->lineno;
debug_end_offset = ((expr_ty) conversion->result)->col_offset;
debug_metadata = conversion->metadata;
}
else if (format) {
debug_end_line = ((expr_ty) format->result)->lineno;
debug_end_offset = ((expr_ty) format->result)->col_offset + 1;
debug_metadata = format->metadata;
}
else {
debug_end_line = end_lineno;
debug_end_offset = end_col_offset;
debug_metadata = closing_brace->metadata;
}
expr_ty debug_text = _PyAST_Constant(debug_metadata, NULL, lineno, col_offset + 1, debug_end_line,
debug_end_offset - 1, p->arena);
if (!debug_text) {
return NULL;
}
asdl_expr_seq *values = _Py_asdl_expr_seq_new(2, arena);
if (values == NULL) {
return NULL;
}
asdl_seq_SET(values, 0, debug_text);
asdl_seq_SET(values, 1, formatted_value);
return _PyAST_JoinedStr(values, lineno, col_offset, debug_end_line, debug_end_offset, p->arena);
}
else {
if (!debug) {
return formatted_value;
}
/* Find the non whitespace token after the "=" */
int debug_end_line, debug_end_offset;
PyObject *debug_metadata;
if (conversion) {
debug_end_line = ((expr_ty) conversion->result)->lineno;
debug_end_offset = ((expr_ty) conversion->result)->col_offset;
debug_metadata = conversion->metadata;
}
else if (format) {
debug_end_line = ((expr_ty) format->result)->lineno;
debug_end_offset = ((expr_ty) format->result)->col_offset + 1;
debug_metadata = format->metadata;
}
else {
debug_end_line = end_lineno;
debug_end_offset = end_col_offset;
debug_metadata = closing_brace->metadata;
}
expr_ty debug_text = _PyAST_Constant(debug_metadata, NULL, lineno, col_offset + 1, debug_end_line,
debug_end_offset - 1, p->arena);
if (!debug_text) {
return NULL;
}
asdl_expr_seq *values = _Py_asdl_expr_seq_new(2, arena);
asdl_seq_SET(values, 0, debug_text);
asdl_seq_SET(values, 1, formatted_value);
return _PyAST_JoinedStr(values, lineno, col_offset, debug_end_line, debug_end_offset, p->arena);
}
expr_ty
_PyPegen_concatenate_strings(Parser *p, asdl_expr_seq *strings,
int lineno, int col_offset, int end_lineno,
int end_col_offset, PyArena *arena)
static expr_ty
_build_concatenated_bytes(Parser *p, asdl_expr_seq *strings, int lineno,
int col_offset, int end_lineno, int end_col_offset,
PyArena *arena)
{
Py_ssize_t len = asdl_seq_LEN(strings);
assert(len > 0);
int f_string_found = 0;
int unicode_string_found = 0;
int bytes_found = 0;
PyObject* res = Py_GetConstant(Py_CONSTANT_EMPTY_BYTES);
Py_ssize_t i = 0;
Py_ssize_t n_flattened_elements = 0;
for (i = 0; i < len; i++) {
/* Bytes literals never get a kind, but just for consistency
since they are represented as Constant nodes, we'll mirror
the same behavior as unicode strings for determining the
kind. */
PyObject* kind = asdl_seq_GET(strings, 0)->v.Constant.kind;
for (Py_ssize_t i = 0; i < len; i++) {
expr_ty elem = asdl_seq_GET(strings, i);
switch(elem->kind) {
case Constant_kind:
if (PyBytes_CheckExact(elem->v.Constant.value)) {
bytes_found = 1;
} else {
unicode_string_found = 1;
}
n_flattened_elements++;
break;
case JoinedStr_kind:
n_flattened_elements += asdl_seq_LEN(elem->v.JoinedStr.values);
f_string_found = 1;
break;
default:
n_flattened_elements++;
f_string_found = 1;
break;
}
PyBytes_Concat(&res, elem->v.Constant.value);
}
if (!res || _PyArena_AddPyObject(arena, res) < 0) {
Py_XDECREF(res);
return NULL;
}
return _PyAST_Constant(res, kind, lineno, col_offset, end_lineno, end_col_offset, p->arena);
}
if ((unicode_string_found || f_string_found) && bytes_found) {
RAISE_SYNTAX_ERROR("cannot mix bytes and nonbytes literals");
static expr_ty
_build_concatenated_unicode(Parser *p, asdl_expr_seq *strings, int lineno,
int col_offset, int end_lineno, int end_col_offset,
PyArena *arena)
{
Py_ssize_t len = asdl_seq_LEN(strings);
assert(len > 1);
expr_ty first = asdl_seq_GET(strings, 0);
/* When a string is getting concatenated, the kind of the string
is determined by the first string in the concatenation
sequence.
u"abc" "def" -> u"abcdef"
"abc" u"abc" -> "abcabc" */
PyObject *kind = first->v.Constant.kind;
PyUnicodeWriter *writer = PyUnicodeWriter_Create(0);
if (writer == NULL) {
return NULL;
}
if (bytes_found) {
PyObject* res = Py_GetConstant(Py_CONSTANT_EMPTY_BYTES);
for (Py_ssize_t i = 0; i < len; i++) {
expr_ty current_elem = asdl_seq_GET(strings, i);
assert(current_elem->kind == Constant_kind);
/* Bytes literals never get a kind, but just for consistency
since they are represented as Constant nodes, we'll mirror
the same behavior as unicode strings for determining the
kind. */
PyObject* kind = asdl_seq_GET(strings, 0)->v.Constant.kind;
for (i = 0; i < len; i++) {
expr_ty elem = asdl_seq_GET(strings, i);
PyBytes_Concat(&res, elem->v.Constant.value);
}
if (!res || _PyArena_AddPyObject(arena, res) < 0) {
Py_XDECREF(res);
if (PyUnicodeWriter_WriteStr(writer,
current_elem->v.Constant.value)) {
PyUnicodeWriter_Discard(writer);
return NULL;
}
return _PyAST_Constant(res, kind, lineno, col_offset, end_lineno, end_col_offset, p->arena);
}
if (!f_string_found && len == 1) {
return asdl_seq_GET(strings, 0);
PyObject *final = PyUnicodeWriter_Finish(writer);
if (final == NULL) {
return NULL;
}
if (_PyArena_AddPyObject(p->arena, final) < 0) {
Py_DECREF(final);
return NULL;
}
return _PyAST_Constant(final, kind, lineno, col_offset,
end_lineno, end_col_offset, arena);
}
static asdl_expr_seq *
_build_concatenated_str(Parser *p, asdl_expr_seq *strings,
int lineno, int col_offset, int end_lineno,
int end_col_offset, PyArena *arena)
{
Py_ssize_t len = asdl_seq_LEN(strings);
assert(len > 0);
Py_ssize_t n_flattened_elements = 0;
for (Py_ssize_t i = 0; i < len; i++) {
expr_ty elem = asdl_seq_GET(strings, i);
switch(elem->kind) {
case JoinedStr_kind:
n_flattened_elements += asdl_seq_LEN(elem->v.JoinedStr.values);
break;
case TemplateStr_kind:
n_flattened_elements += asdl_seq_LEN(elem->v.TemplateStr.values);
break;
default:
n_flattened_elements++;
break;
}
}
asdl_expr_seq* flattened = _Py_asdl_expr_seq_new(n_flattened_elements, p->arena);
if (flattened == NULL) {
@ -1574,12 +1700,11 @@ _PyPegen_concatenate_strings(Parser *p, asdl_expr_seq *strings,
/* build flattened list */
Py_ssize_t current_pos = 0;
Py_ssize_t j = 0;
for (i = 0; i < len; i++) {
for (Py_ssize_t i = 0; i < len; i++) {
expr_ty elem = asdl_seq_GET(strings, i);
switch(elem->kind) {
case JoinedStr_kind:
for (j = 0; j < asdl_seq_LEN(elem->v.JoinedStr.values); j++) {
for (Py_ssize_t j = 0; j < asdl_seq_LEN(elem->v.JoinedStr.values); j++) {
expr_ty subvalue = asdl_seq_GET(elem->v.JoinedStr.values, j);
if (subvalue == NULL) {
return NULL;
@ -1587,6 +1712,15 @@ _PyPegen_concatenate_strings(Parser *p, asdl_expr_seq *strings,
asdl_seq_SET(flattened, current_pos++, subvalue);
}
break;
case TemplateStr_kind:
for (Py_ssize_t j = 0; j < asdl_seq_LEN(elem->v.TemplateStr.values); j++) {
expr_ty subvalue = asdl_seq_GET(elem->v.TemplateStr.values, j);
if (subvalue == NULL) {
return NULL;
}
asdl_seq_SET(flattened, current_pos++, subvalue);
}
break;
default:
asdl_seq_SET(flattened, current_pos++, elem);
break;
@ -1596,13 +1730,13 @@ _PyPegen_concatenate_strings(Parser *p, asdl_expr_seq *strings,
/* calculate folded element count */
Py_ssize_t n_elements = 0;
int prev_is_constant = 0;
for (i = 0; i < n_flattened_elements; i++) {
for (Py_ssize_t i = 0; i < n_flattened_elements; i++) {
expr_ty elem = asdl_seq_GET(flattened, i);
/* The concatenation of a FormattedValue and an empty Constant should
lead to the FormattedValue itself. Thus, we will not take any empty
constants into account, just as in `_PyPegen_joined_str` */
if (f_string_found && elem->kind == Constant_kind &&
if (elem->kind == Constant_kind &&
PyUnicode_CheckExact(elem->v.Constant.value) &&
PyUnicode_GET_LENGTH(elem->v.Constant.value) == 0)
continue;
@ -1620,7 +1754,7 @@ _PyPegen_concatenate_strings(Parser *p, asdl_expr_seq *strings,
/* build folded list */
current_pos = 0;
for (i = 0; i < n_flattened_elements; i++) {
for (Py_ssize_t i = 0; i < n_flattened_elements; i++) {
expr_ty elem = asdl_seq_GET(flattened, i);
/* if the current elem and the following are constants,
@ -1643,6 +1777,7 @@ _PyPegen_concatenate_strings(Parser *p, asdl_expr_seq *strings,
return NULL;
}
expr_ty last_elem = elem;
Py_ssize_t j;
for (j = i; j < n_flattened_elements; j++) {
expr_ty current_elem = asdl_seq_GET(flattened, j);
if (current_elem->kind == Constant_kind) {
@ -1676,8 +1811,7 @@ _PyPegen_concatenate_strings(Parser *p, asdl_expr_seq *strings,
}
/* Drop all empty contanst strings */
if (f_string_found &&
PyUnicode_CheckExact(elem->v.Constant.value) &&
if (PyUnicode_CheckExact(elem->v.Constant.value) &&
PyUnicode_GET_LENGTH(elem->v.Constant.value) == 0) {
continue;
}
@ -1686,15 +1820,95 @@ _PyPegen_concatenate_strings(Parser *p, asdl_expr_seq *strings,
asdl_seq_SET(values, current_pos++, elem);
}
if (!f_string_found) {
assert(n_elements == 1);
expr_ty elem = asdl_seq_GET(values, 0);
assert(elem->kind == Constant_kind);
return elem;
assert(current_pos == n_elements);
return values;
}
static expr_ty
_build_concatenated_joined_str(Parser *p, asdl_expr_seq *strings,
int lineno, int col_offset, int end_lineno,
int end_col_offset, PyArena *arena)
{
asdl_expr_seq *values = _build_concatenated_str(p, strings, lineno,
col_offset, end_lineno, end_col_offset, arena);
return _PyAST_JoinedStr(values, lineno, col_offset, end_lineno, end_col_offset, p->arena);
}
static expr_ty
_build_concatenated_template_str(Parser *p, asdl_expr_seq *strings,
int lineno, int col_offset, int end_lineno,
int end_col_offset, PyArena *arena)
{
asdl_expr_seq *values = _build_concatenated_str(p, strings, lineno,
col_offset, end_lineno, end_col_offset, arena);
return _PyAST_TemplateStr(values, lineno, col_offset, end_lineno,
end_col_offset, arena);
}
expr_ty
_PyPegen_concatenate_strings(Parser *p, asdl_expr_seq *strings,
int lineno, int col_offset, int end_lineno,
int end_col_offset, PyArena *arena)
{
Py_ssize_t len = asdl_seq_LEN(strings);
assert(len > 0);
int t_string_found = 0;
int f_string_found = 0;
int unicode_string_found = 0;
int bytes_found = 0;
Py_ssize_t i = 0;
for (i = 0; i < len; i++) {
expr_ty elem = asdl_seq_GET(strings, i);
switch(elem->kind) {
case Constant_kind:
if (PyBytes_CheckExact(elem->v.Constant.value)) {
bytes_found = 1;
} else {
unicode_string_found = 1;
}
break;
case JoinedStr_kind:
f_string_found = 1;
break;
case TemplateStr_kind:
t_string_found = 1;
break;
default:
f_string_found = 1;
break;
}
}
assert(current_pos == n_elements);
return _PyAST_JoinedStr(values, lineno, col_offset, end_lineno, end_col_offset, p->arena);
// Cannot mix unicode and bytes
if ((unicode_string_found || f_string_found || t_string_found) && bytes_found) {
RAISE_SYNTAX_ERROR("cannot mix bytes and nonbytes literals");
return NULL;
}
// If it's only bytes or only unicode string, do a simple concat
if (!f_string_found && !t_string_found) {
if (len == 1) {
return asdl_seq_GET(strings, 0);
}
else if (bytes_found) {
return _build_concatenated_bytes(p, strings, lineno, col_offset,
end_lineno, end_col_offset, arena);
}
else {
return _build_concatenated_unicode(p, strings, lineno, col_offset,
end_lineno, end_col_offset, arena);
}
}
if (t_string_found) {
return _build_concatenated_template_str(p, strings, lineno,
col_offset, end_lineno, end_col_offset, arena);
}
return _build_concatenated_joined_str(p, strings, lineno,
col_offset, end_lineno, end_col_offset, arena);
}
stmt_ty

8
Parser/lexer/buffer.c
View File

@ -13,8 +13,8 @@ _PyLexer_remember_fstring_buffers(struct tok_state *tok)
for (index = tok->tok_mode_stack_index; index >= 0; --index) {
mode = &(tok->tok_mode_stack[index]);
mode->f_string_start_offset = mode->f_string_start - tok->buf;
mode->f_string_multi_line_start_offset = mode->f_string_multi_line_start - tok->buf;
mode->start_offset = mode->start - tok->buf;
mode->multi_line_start_offset = mode->multi_line_start - tok->buf;
}
}
@ -27,8 +27,8 @@ _PyLexer_restore_fstring_buffers(struct tok_state *tok)
for (index = tok->tok_mode_stack_index; index >= 0; --index) {
mode = &(tok->tok_mode_stack[index]);
mode->f_string_start = tok->buf + mode->f_string_start_offset;
mode->f_string_multi_line_start = tok->buf + mode->f_string_multi_line_start_offset;
mode->start = tok->buf + mode->start_offset;
mode->multi_line_start = tok->buf + mode->multi_line_start_offset;
}
}

143
Parser/lexer/lexer.c
View File

@ -38,6 +38,9 @@ static inline tokenizer_mode* TOK_NEXT_MODE(struct tok_state* tok) {
#define TOK_NEXT_MODE(tok) (&(tok->tok_mode_stack[++tok->tok_mode_stack_index]))
#endif
#define FTSTRING_MIDDLE(tok_mode) (tok_mode->string_kind == TSTRING ? TSTRING_MIDDLE : FSTRING_MIDDLE)
#define FTSTRING_END(tok_mode) (tok_mode->string_kind == TSTRING ? TSTRING_END : FSTRING_END)
#define TOK_GET_STRING_PREFIX(tok) (TOK_GET_MODE(tok)->string_kind == TSTRING ? 't' : 'f')
#define MAKE_TOKEN(token_type) _PyLexer_token_setup(tok, token, token_type, p_start, p_end)
#define MAKE_TYPE_COMMENT_TOKEN(token_type, col_offset, end_col_offset) (\
_PyLexer_type_comment_token_setup(tok, token, token_type, col_offset, end_col_offset, p_start, p_end))
@ -108,12 +111,12 @@ tok_backup(struct tok_state *tok, int c)
}
static int
set_fstring_expr(struct tok_state* tok, struct token *token, char c) {
set_ftstring_expr(struct tok_state* tok, struct token *token, char c) {
assert(token != NULL);
assert(c == '}' || c == ':' || c == '!');
tokenizer_mode *tok_mode = TOK_GET_MODE(tok);
if (!tok_mode->f_string_debug || token->metadata) {
if (!(tok_mode->in_debug || tok_mode->string_kind == TSTRING) || token->metadata) {
return 0;
}
PyObject *res = NULL;
@ -173,7 +176,7 @@ set_fstring_expr(struct tok_state* tok, struct token *token, char c) {
}
int
_PyLexer_update_fstring_expr(struct tok_state *tok, char cur)
_PyLexer_update_ftstring_expr(struct tok_state *tok, char cur)
{
assert(tok->cur != NULL);
@ -643,13 +646,13 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
nonascii = 0;
if (is_potential_identifier_start(c)) {
/* Process the various legal combinations of b"", r"", u"", and f"". */
int saw_b = 0, saw_r = 0, saw_u = 0, saw_f = 0;
int saw_b = 0, saw_r = 0, saw_u = 0, saw_f = 0, saw_t = 0;
while (1) {
if (!(saw_b || saw_u || saw_f) && (c == 'b' || c == 'B'))
if (!(saw_b || saw_u || saw_f || saw_t) && (c == 'b' || c == 'B'))
saw_b = 1;
/* Since this is a backwards compatibility support literal we don't
want to support it in arbitrary order like byte literals. */
else if (!(saw_b || saw_u || saw_r || saw_f)
else if (!(saw_b || saw_u || saw_r || saw_f || saw_t)
&& (c == 'u'|| c == 'U')) {
saw_u = 1;
}
@ -657,15 +660,18 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
else if (!(saw_r || saw_u) && (c == 'r' || c == 'R')) {
saw_r = 1;
}
else if (!(saw_f || saw_b || saw_u) && (c == 'f' || c == 'F')) {
else if (!(saw_f || saw_b || saw_u || saw_t) && (c == 'f' || c == 'F')) {
saw_f = 1;
}
else if (!(saw_t || saw_b || saw_u || saw_f) && (c == 't' || c == 'T')) {
saw_t = 1;
}
else {
break;
}
c = tok_nextc(tok);
if (c == '"' || c == '\'') {
if (saw_f) {
if (saw_f || saw_t) {
goto f_string_quote;
}
goto letter_quote;
@ -939,7 +945,9 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
}
f_string_quote:
if (((Py_TOLOWER(*tok->start) == 'f' || Py_TOLOWER(*tok->start) == 'r') && (c == '\'' || c == '"'))) {
if (((Py_TOLOWER(*tok->start) == 'f' || Py_TOLOWER(*tok->start) == 'r' || Py_TOLOWER(*tok->start) == 't')
&& (c == '\'' || c == '"'))) {
int quote = c;
int quote_size = 1; /* 1 or 3 */
@ -971,39 +979,49 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
p_start = tok->start;
p_end = tok->cur;
if (tok->tok_mode_stack_index + 1 >= MAXFSTRINGLEVEL) {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "too many nested f-strings"));
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "too many nested f-strings or t-strings"));
}
tokenizer_mode *the_current_tok = TOK_NEXT_MODE(tok);
the_current_tok->kind = TOK_FSTRING_MODE;
the_current_tok->f_string_quote = quote;
the_current_tok->f_string_quote_size = quote_size;
the_current_tok->f_string_start = tok->start;
the_current_tok->f_string_multi_line_start = tok->line_start;
the_current_tok->f_string_line_start = tok->lineno;
the_current_tok->f_string_start_offset = -1;
the_current_tok->f_string_multi_line_start_offset = -1;
the_current_tok->quote = quote;
the_current_tok->quote_size = quote_size;
the_current_tok->start = tok->start;
the_current_tok->multi_line_start = tok->line_start;
the_current_tok->first_line = tok->lineno;
the_current_tok->start_offset = -1;
the_current_tok->multi_line_start_offset = -1;
the_current_tok->last_expr_buffer = NULL;
the_current_tok->last_expr_size = 0;
the_current_tok->last_expr_end = -1;
the_current_tok->in_format_spec = 0;
the_current_tok->f_string_debug = 0;
the_current_tok->in_debug = 0;
enum string_kind_t string_kind = FSTRING;
switch (*tok->start) {
case 'T':
case 't':
the_current_tok->raw = Py_TOLOWER(*(tok->start + 1)) == 'r';
string_kind = TSTRING;
break;
case 'F':
case 'f':
the_current_tok->f_string_raw = Py_TOLOWER(*(tok->start + 1)) == 'r';
the_current_tok->raw = Py_TOLOWER(*(tok->start + 1)) == 'r';
break;
case 'R':
case 'r':
the_current_tok->f_string_raw = 1;
the_current_tok->raw = 1;
if (Py_TOLOWER(*(tok->start + 1)) == 't') {
string_kind = TSTRING;
}
break;
default:
Py_UNREACHABLE();
}
the_current_tok->string_kind = string_kind;
the_current_tok->curly_bracket_depth = 0;
the_current_tok->curly_bracket_expr_start_depth = -1;
return MAKE_TOKEN(FSTRING_START);
return string_kind == TSTRING ? MAKE_TOKEN(TSTRING_START) : MAKE_TOKEN(FSTRING_START);
}
letter_quote:
@ -1063,9 +1081,10 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
* and if it is, then this must be a missing '}' token
* so raise the proper error */
tokenizer_mode *the_current_tok = TOK_GET_MODE(tok);
if (the_current_tok->f_string_quote == quote &&
the_current_tok->f_string_quote_size == quote_size) {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "f-string: expecting '}'", start));
if (the_current_tok->quote == quote &&
the_current_tok->quote_size == quote_size) {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok,
"%c-string: expecting '}'", TOK_GET_STRING_PREFIX(tok)));
}
}
@ -1136,12 +1155,12 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
int cursor = current_tok->curly_bracket_depth - (c != '{');
int in_format_spec = current_tok->in_format_spec;
int cursor_in_format_with_debug =
cursor == 1 && (current_tok->f_string_debug || in_format_spec);
cursor == 1 && (current_tok->in_debug || in_format_spec);
int cursor_valid = cursor == 0 || cursor_in_format_with_debug;
if ((cursor_valid) && !_PyLexer_update_fstring_expr(tok, c)) {
if ((cursor_valid) && !_PyLexer_update_ftstring_expr(tok, c)) {
return MAKE_TOKEN(ENDMARKER);
}
if ((cursor_valid) && c != '{' && set_fstring_expr(tok, token, c)) {
if ((cursor_valid) && c != '{' && set_ftstring_expr(tok, token, c)) {
return MAKE_TOKEN(ERRORTOKEN);
}
@ -1194,7 +1213,8 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
case ']':
case '}':
if (INSIDE_FSTRING(tok) && !current_tok->curly_bracket_depth && c == '}') {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "f-string: single '}' is not allowed"));
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok,
"%c-string: single '}' is not allowed", TOK_GET_STRING_PREFIX(tok)));
}
if (!tok->tok_extra_tokens && !tok->level) {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "unmatched '%c'", c));
@ -1214,7 +1234,8 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
assert(current_tok->curly_bracket_depth >= 0);
int previous_bracket = current_tok->curly_bracket_depth - 1;
if (previous_bracket == current_tok->curly_bracket_expr_start_depth) {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "f-string: unmatched '%c'", c));
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok,
"%c-string: unmatched '%c'", TOK_GET_STRING_PREFIX(tok), c));
}
}
if (tok->parenlinenostack[tok->level] != tok->lineno) {
@ -1235,13 +1256,14 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
if (INSIDE_FSTRING(tok)) {
current_tok->curly_bracket_depth--;
if (current_tok->curly_bracket_depth < 0) {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "f-string: unmatched '%c'", c));
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "%c-string: unmatched '%c'",
TOK_GET_STRING_PREFIX(tok), c));
}
if (c == '}' && current_tok->curly_bracket_depth == current_tok->curly_bracket_expr_start_depth) {
current_tok->curly_bracket_expr_start_depth--;
current_tok->kind = TOK_FSTRING_MODE;
current_tok->in_format_spec = 0;
current_tok->f_string_debug = 0;
current_tok->in_debug = 0;
}
}
break;
@ -1254,7 +1276,7 @@ tok_get_normal_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct t
}
if( c == '=' && INSIDE_FSTRING_EXPR(current_tok)) {
current_tok->f_string_debug = 1;
current_tok->in_debug = 1;
}
/* Punctuation character */
@ -1285,7 +1307,8 @@ tok_get_fstring_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct
if (peek1 != '{') {
current_tok->curly_bracket_expr_start_depth++;
if (current_tok->curly_bracket_expr_start_depth >= MAX_EXPR_NESTING) {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "f-string: expressions nested too deeply"));
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok,
"%c-string: expressions nested too deeply", TOK_GET_STRING_PREFIX(tok)));
}
TOK_GET_MODE(tok)->kind = TOK_REGULAR_MODE;
return tok_get_normal_mode(tok, current_tok, token);
@ -1296,9 +1319,9 @@ tok_get_fstring_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct
}
// Check if we are at the end of the string
for (int i = 0; i < current_tok->f_string_quote_size; i++) {
for (int i = 0; i < current_tok->quote_size; i++) {
int quote = tok_nextc(tok);
if (quote != current_tok->f_string_quote) {
if (quote != current_tok->quote) {
tok_backup(tok, quote);
goto f_string_middle;
}
@ -1314,14 +1337,14 @@ tok_get_fstring_mode(struct tok_state *tok, tokenizer_mode* current_tok, struct
p_start = tok->start;
p_end = tok->cur;
tok->tok_mode_stack_index--;
return MAKE_TOKEN(FSTRING_END);
return MAKE_TOKEN(FTSTRING_END(current_tok));
f_string_middle:
// TODO: This is a bit of a hack, but it works for now. We need to find a better way to handle
// this.
tok->multi_line_start = tok->line_start;
while (end_quote_size != current_tok->f_string_quote_size) {
while (end_quote_size != current_tok->quote_size) {
int c = tok_nextc(tok);
if (tok->done == E_ERROR || tok->done == E_DECODE) {
return MAKE_TOKEN(ERRORTOKEN);
@ -1332,7 +1355,7 @@ f_string_middle:
INSIDE_FSTRING_EXPR(current_tok)
);
if (c == EOF || (current_tok->f_string_quote_size == 1 && c == '\n')) {
if (c == EOF || (current_tok->quote_size == 1 && c == '\n')) {
if (tok->decoding_erred) {
return MAKE_TOKEN(ERRORTOKEN);
}
@ -1341,7 +1364,7 @@ f_string_middle:
// it means that the format spec ends here and we should
// return to the regular mode.
if (in_format_spec && c == '\n') {
if (current_tok->f_string_quote_size == 1) {
if (current_tok->quote_size == 1) {
return MAKE_TOKEN(
_PyTokenizer_syntaxerror(
tok,
@ -1354,25 +1377,26 @@ f_string_middle:
current_tok->in_format_spec = 0;
p_start = tok->start;
p_end = tok->cur;
return MAKE_TOKEN(FSTRING_MIDDLE);
return MAKE_TOKEN(FTSTRING_MIDDLE(current_tok));
}
assert(tok->multi_line_start != NULL);
// shift the tok_state's location into
// the start of string, and report the error
// from the initial quote character
tok->cur = (char *)current_tok->f_string_start;
tok->cur = (char *)current_tok->start;
tok->cur++;
tok->line_start = current_tok->f_string_multi_line_start;
tok->line_start = current_tok->multi_line_start;
int start = tok->lineno;
tokenizer_mode *the_current_tok = TOK_GET_MODE(tok);
tok->lineno = the_current_tok->f_string_line_start;
tok->lineno = the_current_tok->first_line;
if (current_tok->f_string_quote_size == 3) {
if (current_tok->quote_size == 3) {
_PyTokenizer_syntaxerror(tok,
"unterminated triple-quoted f-string literal"
" (detected at line %d)", start);
"unterminated triple-quoted %c-string literal"
" (detected at line %d)",
TOK_GET_STRING_PREFIX(tok), start);
if (c != '\n') {
tok->done = E_EOFS;
}
@ -1380,12 +1404,12 @@ f_string_middle:
}
else {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok,
"unterminated f-string literal (detected at"
" line %d)", start));
"unterminated %c-string literal (detected at"
" line %d)", TOK_GET_STRING_PREFIX(tok), start));
}
}
if (c == current_tok->f_string_quote) {
if (c == current_tok->quote) {
end_quote_size += 1;
continue;
} else {
@ -1393,7 +1417,7 @@ f_string_middle:
}
if (c == '{') {
if (!_PyLexer_update_fstring_expr(tok, c)) {
if (!_PyLexer_update_ftstring_expr(tok, c)) {
return MAKE_TOKEN(ENDMARKER);
}
int peek = tok_nextc(tok);
@ -1402,7 +1426,8 @@ f_string_middle:
tok_backup(tok, c);
current_tok->curly_bracket_expr_start_depth++;
if (current_tok->curly_bracket_expr_start_depth >= MAX_EXPR_NESTING) {
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok, "f-string: expressions nested too deeply"));
return MAKE_TOKEN(_PyTokenizer_syntaxerror(tok,
"%c-string: expressions nested too deeply", TOK_GET_STRING_PREFIX(tok)));
}
TOK_GET_MODE(tok)->kind = TOK_REGULAR_MODE;
current_tok->in_format_spec = 0;
@ -1412,12 +1437,12 @@ f_string_middle:
p_start = tok->start;
p_end = tok->cur - 1;
}
return MAKE_TOKEN(FSTRING_MIDDLE);
return MAKE_TOKEN(FTSTRING_MIDDLE(current_tok));
} else if (c == '}') {
if (unicode_escape) {
p_start = tok->start;
p_end = tok->cur;
return MAKE_TOKEN(FSTRING_MIDDLE);
return MAKE_TOKEN(FTSTRING_MIDDLE(current_tok));
}
int peek = tok_nextc(tok);
@ -1437,7 +1462,7 @@ f_string_middle:
p_start = tok->start;
p_end = tok->cur;
}
return MAKE_TOKEN(FSTRING_MIDDLE);
return MAKE_TOKEN(FTSTRING_MIDDLE(current_tok));
} else if (c == '\\') {
int peek = tok_nextc(tok);
if (peek == '\r') {
@ -1447,7 +1472,7 @@ f_string_middle:
// brace. We have to restore and return the control back
// to the loop for the next iteration.
if (peek == '{' || peek == '}') {
if (!current_tok->f_string_raw) {
if (!current_tok->raw) {
if (_PyTokenizer_warn_invalid_escape_sequence(tok, peek)) {
return MAKE_TOKEN(ERRORTOKEN);
}
@ -1456,7 +1481,7 @@ f_string_middle:
continue;
}
if (!current_tok->f_string_raw) {
if (!current_tok->raw) {
if (peek == 'N') {
/* Handle named unicode escapes (\N{BULLET}) */
peek = tok_nextc(tok);
@ -1474,12 +1499,12 @@ f_string_middle:
// Backup the f-string quotes to emit a final FSTRING_MIDDLE and
// add the quotes to the FSTRING_END in the next tokenizer iteration.
for (int i = 0; i < current_tok->f_string_quote_size; i++) {
tok_backup(tok, current_tok->f_string_quote);
for (int i = 0; i < current_tok->quote_size; i++) {
tok_backup(tok, current_tok->quote);
}
p_start = tok->start;
p_end = tok->cur;
return MAKE_TOKEN(FSTRING_MIDDLE);
return MAKE_TOKEN(FTSTRING_MIDDLE(current_tok));
}
static int

2
Parser/lexer/lexer.h
View File

@ -3,7 +3,7 @@
#include "state.h"
int _PyLexer_update_fstring_expr(struct tok_state *tok, char cur);
int _PyLexer_update_ftstring_expr(struct tok_state *tok, char cur);
int _PyTokenizer_Get(struct tok_state *, struct token *);

2
Parser/lexer/state.c
View File

@ -54,7 +54,7 @@ _PyTokenizer_tok_new(void)
tok->tok_extra_tokens = 0;
tok->comment_newline = 0;
tok->implicit_newline = 0;
tok->tok_mode_stack[0] = (tokenizer_mode){.kind =TOK_REGULAR_MODE, .f_string_quote='\0', .f_string_quote_size = 0, .f_string_debug=0};
tok->tok_mode_stack[0] = (tokenizer_mode){.kind =TOK_REGULAR_MODE, .quote='\0', .quote_size = 0, .in_debug=0};
tok->tok_mode_stack_index = 0;
#ifdef Py_DEBUG
tok->debug = _Py_GetConfig()->parser_debug;

25
Parser/lexer/state.h
View File

@ -36,6 +36,11 @@ enum tokenizer_mode_kind_t {
TOK_FSTRING_MODE,
};
enum string_kind_t {
FSTRING,
TSTRING,
};
#define MAX_EXPR_NESTING 3
typedef struct _tokenizer_mode {
@ -44,21 +49,23 @@ typedef struct _tokenizer_mode {
int curly_bracket_depth;
int curly_bracket_expr_start_depth;
char f_string_quote;
int f_string_quote_size;
int f_string_raw;
const char* f_string_start;
const char* f_string_multi_line_start;
int f_string_line_start;
char quote;
int quote_size;
int raw;
const char* start;
const char* multi_line_start;
int first_line;
Py_ssize_t f_string_start_offset;
Py_ssize_t f_string_multi_line_start_offset;
Py_ssize_t start_offset;
Py_ssize_t multi_line_start_offset;
Py_ssize_t last_expr_size;
Py_ssize_t last_expr_end;
char* last_expr_buffer;
int f_string_debug;
int in_debug;
int in_format_spec;
enum string_kind_t string_kind;
} tokenizer_mode;
/* Tokenizer state */

7498
Parser/parser.c generated
View File

File diff suppressed because it is too large Load Diff

12
Parser/pegen.h
View File

@ -5,6 +5,8 @@
#include <pycore_ast.h>
#include <pycore_token.h>
#include "lexer/state.h"
#if 0
#define PyPARSE_YIELD_IS_KEYWORD 0x0001
#endif
@ -24,6 +26,9 @@
#define CURRENT_POS (-5)
#define TOK_GET_MODE(tok) (&(tok->tok_mode_stack[tok->tok_mode_stack_index]))
#define TOK_GET_STRING_PREFIX(tok) (TOK_GET_MODE(tok)->string_kind == TSTRING ? 't' : 'f')
typedef struct _memo {
int type;
void *node;
@ -327,6 +332,10 @@ StarEtc *_PyPegen_star_etc(Parser *, arg_ty, asdl_seq *, arg_ty);
arguments_ty _PyPegen_make_arguments(Parser *, asdl_arg_seq *, SlashWithDefault *,
asdl_arg_seq *, asdl_seq *, StarEtc *);
arguments_ty _PyPegen_empty_arguments(Parser *);
expr_ty _PyPegen_template_str(Parser *p, Token *a, asdl_expr_seq *raw_expressions, Token *b);
expr_ty _PyPegen_joined_str(Parser *p, Token *a, asdl_expr_seq *raw_expressions, Token *b);
expr_ty _PyPegen_interpolation(Parser *, expr_ty, Token *, ResultTokenWithMetadata *, ResultTokenWithMetadata *, Token *,
int, int, int, int, PyArena *);
expr_ty _PyPegen_formatted_value(Parser *, expr_ty, Token *, ResultTokenWithMetadata *, ResultTokenWithMetadata *, Token *,
int, int, int, int, PyArena *);
AugOperator *_PyPegen_augoperator(Parser*, operator_ty type);
@ -371,9 +380,6 @@ void *_PyPegen_run_parser(Parser *);
mod_ty _PyPegen_run_parser_from_string(const char *, int, PyObject *, PyCompilerFlags *, PyArena *);
asdl_stmt_seq *_PyPegen_interactive_exit(Parser *);
// TODO: move to the correct place in this file
expr_ty _PyPegen_joined_str(Parser *p, Token* a, asdl_expr_seq* expr, Token*b);
// Generated function in parse.c - function definition in python.gram
void *_PyPegen_parse(Parser *);

3
Parser/string_parser.c
View File

@ -19,7 +19,8 @@ warn_invalid_escape_sequence(Parser *p, const char* buffer, const char *first_in
return 0;
}
unsigned char c = (unsigned char)*first_invalid_escape;
if ((t->type == FSTRING_MIDDLE || t->type == FSTRING_END) && (c == '{' || c == '}')) {
if ((t->type == FSTRING_MIDDLE || t->type == FSTRING_END || t->type == TSTRING_MIDDLE || t->type == TSTRING_END)
&& (c == '{' || c == '}')) {
// in this case the tokenizer has already emitted a warning,
// see Parser/tokenizer/helpers.c:warn_invalid_escape_sequence
return 0;

3
Parser/token.c generated
View File

@ -68,6 +68,9 @@ const char * const _PyParser_TokenNames[] = {
"FSTRING_START",
"FSTRING_MIDDLE",
"FSTRING_END",
"TSTRING_START",
"TSTRING_MIDDLE",
"TSTRING_END",
"COMMENT",
"NL",
"<ERRORTOKEN>",

4
Parser/tokenizer/file_tokenizer.c
View File

@ -275,7 +275,7 @@ tok_underflow_interactive(struct tok_state *tok) {
return 0;
}
if (tok->tok_mode_stack_index && !_PyLexer_update_fstring_expr(tok, 0)) {
if (tok->tok_mode_stack_index && !_PyLexer_update_ftstring_expr(tok, 0)) {
return 0;
}
return 1;
@ -322,7 +322,7 @@ tok_underflow_file(struct tok_state *tok) {
tok->implicit_newline = 1;
}
if (tok->tok_mode_stack_index && !_PyLexer_update_fstring_expr(tok, 0)) {
if (tok->tok_mode_stack_index && !_PyLexer_update_ftstring_expr(tok, 0)) {
return 0;
}

2
Parser/tokenizer/readline_tokenizer.c
View File

@ -90,7 +90,7 @@ tok_underflow_readline(struct tok_state* tok) {
tok->implicit_newline = 1;
}
if (tok->tok_mode_stack_index && !_PyLexer_update_fstring_expr(tok, 0)) {
if (tok->tok_mode_stack_index && !_PyLexer_update_ftstring_expr(tok, 0)) {
return 0;
}

26
Programs/test_frozenmain.h generated
View File

@ -1,19 +1,19 @@
// Auto-generated by Programs/freeze_test_frozenmain.py
unsigned char M_test_frozenmain[] = {
227,0,0,0,0,0,0,0,0,0,0,0,0,8,0,0,
0,0,0,0,0,243,184,0,0,0,128,0,92,0,80,1,
71,0,114,0,92,0,80,1,71,1,114,1,91,2,32,0,
80,2,50,1,0,0,0,0,0,0,30,0,91,2,32,0,
80,3,91,0,78,6,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,50,2,0,0,0,0,0,0,
30,0,91,1,78,8,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,32,0,50,0,0,0,0,0,
0,0,80,4,43,26,0,0,0,0,0,0,0,0,0,0,
114,5,80,7,15,0,68,24,0,0,114,6,91,2,32,0,
80,5,91,6,11,0,80,6,91,5,91,6,43,26,0,0,
0,0,0,0,0,0,0,0,11,0,48,4,50,1,0,0,
0,0,0,0,30,0,73,26,0,0,8,0,29,0,80,1,
34,0,41,8,233,0,0,0,0,78,122,18,70,114,111,122,
0,0,0,0,0,243,184,0,0,0,128,0,94,0,82,1,
73,0,116,0,94,0,82,1,73,1,116,1,93,2,33,0,
82,2,52,1,0,0,0,0,0,0,31,0,93,2,33,0,
82,3,93,0,80,6,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,52,2,0,0,0,0,0,0,
31,0,93,1,80,8,0,0,0,0,0,0,0,0,0,0,
0,0,0,0,0,0,0,0,33,0,52,0,0,0,0,0,
0,0,82,4,44,26,0,0,0,0,0,0,0,0,0,0,
116,5,82,7,16,0,70,24,0,0,116,6,93,2,33,0,
82,5,93,6,12,0,82,6,93,5,93,6,44,26,0,0,
0,0,0,0,0,0,0,0,12,0,50,4,52,1,0,0,
0,0,0,0,31,0,75,26,0,0,9,0,30,0,82,1,
35,0,41,8,233,0,0,0,0,78,122,18,70,114,111,122,
101,110,32,72,101,108,108,111,32,87,111,114,108,100,122,8,
115,121,115,46,97,114,103,118,218,6,99,111,110,102,105,103,
122,7,99,111,110,102,105,103,32,122,2,58,32,41,5,218,

349
Python/Python-ast.c generated
View File

@ -92,6 +92,7 @@ void _PyAST_Fini(PyInterpreterState *interp)
Py_CLEAR(state->In_singleton);
Py_CLEAR(state->In_type);
Py_CLEAR(state->Interactive_type);
Py_CLEAR(state->Interpolation_type);
Py_CLEAR(state->Invert_singleton);
Py_CLEAR(state->Invert_type);
Py_CLEAR(state->IsNot_singleton);
@ -154,6 +155,7 @@ void _PyAST_Fini(PyInterpreterState *interp)
Py_CLEAR(state->Sub_singleton);
Py_CLEAR(state->Sub_type);
Py_CLEAR(state->Subscript_type);
Py_CLEAR(state->TemplateStr_type);
Py_CLEAR(state->TryStar_type);
Py_CLEAR(state->Try_type);
Py_CLEAR(state->Tuple_type);
@ -259,6 +261,7 @@ void _PyAST_Fini(PyInterpreterState *interp)
Py_CLEAR(state->slice);
Py_CLEAR(state->step);
Py_CLEAR(state->stmt_type);
Py_CLEAR(state->str);
Py_CLEAR(state->subject);
Py_CLEAR(state->tag);
Py_CLEAR(state->target);
@ -357,6 +360,7 @@ static int init_identifiers(struct ast_state *state)
if ((state->simple = PyUnicode_InternFromString("simple")) == NULL) return -1;
if ((state->slice = PyUnicode_InternFromString("slice")) == NULL) return -1;
if ((state->step = PyUnicode_InternFromString("step")) == NULL) return -1;
if ((state->str = PyUnicode_InternFromString("str")) == NULL) return -1;
if ((state->subject = PyUnicode_InternFromString("subject")) == NULL) return -1;
if ((state->tag = PyUnicode_InternFromString("tag")) == NULL) return -1;
if ((state->target = PyUnicode_InternFromString("target")) == NULL) return -1;
@ -619,9 +623,18 @@ static const char * const FormattedValue_fields[]={
"conversion",
"format_spec",
};
static const char * const Interpolation_fields[]={
"value",
"str",
"conversion",
"format_spec",
};
static const char * const JoinedStr_fields[]={
"values",
};
static const char * const TemplateStr_fields[]={
"values",
};
static const char * const Constant_fields[]={
"value",
"kind",
@ -3174,6 +3187,70 @@ add_ast_annotations(struct ast_state *state)
return 0;
}
Py_DECREF(FormattedValue_annotations);
PyObject *Interpolation_annotations = PyDict_New();
if (!Interpolation_annotations) return 0;
{
PyObject *type = state->expr_type;
Py_INCREF(type);
cond = PyDict_SetItemString(Interpolation_annotations, "value", type)
== 0;
Py_DECREF(type);
if (!cond) {
Py_DECREF(Interpolation_annotations);
return 0;
}
}
{
PyObject *type = (PyObject *)&PyBaseObject_Type;
Py_INCREF(type);
cond = PyDict_SetItemString(Interpolation_annotations, "str", type) ==
0;
Py_DECREF(type);
if (!cond) {
Py_DECREF(Interpolation_annotations);
return 0;
}
}
{
PyObject *type = (PyObject *)&PyLong_Type;
Py_INCREF(type);
cond = PyDict_SetItemString(Interpolation_annotations, "conversion",
type) == 0;
Py_DECREF(type);
if (!cond) {
Py_DECREF(Interpolation_annotations);
return 0;
}
}
{
PyObject *type = state->expr_type;
type = _Py_union_type_or(type, Py_None);
cond = type != NULL;
if (!cond) {
Py_DECREF(Interpolation_annotations);
return 0;
}
cond = PyDict_SetItemString(Interpolation_annotations, "format_spec",
type) == 0;
Py_DECREF(type);
if (!cond) {
Py_DECREF(Interpolation_annotations);
return 0;
}
}
cond = PyObject_SetAttrString(state->Interpolation_type, "_field_types",
Interpolation_annotations) == 0;
if (!cond) {
Py_DECREF(Interpolation_annotations);
return 0;
}
cond = PyObject_SetAttrString(state->Interpolation_type, "__annotations__",
Interpolation_annotations) == 0;
if (!cond) {
Py_DECREF(Interpolation_annotations);
return 0;
}
Py_DECREF(Interpolation_annotations);
PyObject *JoinedStr_annotations = PyDict_New();
if (!JoinedStr_annotations) return 0;
{
@ -3204,6 +3281,37 @@ add_ast_annotations(struct ast_state *state)
return 0;
}
Py_DECREF(JoinedStr_annotations);
PyObject *TemplateStr_annotations = PyDict_New();
if (!TemplateStr_annotations) return 0;
{
PyObject *type = state->expr_type;
type = Py_GenericAlias((PyObject *)&PyList_Type, type);
cond = type != NULL;
if (!cond) {
Py_DECREF(TemplateStr_annotations);
return 0;
}
cond = PyDict_SetItemString(TemplateStr_annotations, "values", type) ==
0;
Py_DECREF(type);
if (!cond) {
Py_DECREF(TemplateStr_annotations);
return 0;
}
}
cond = PyObject_SetAttrString(state->TemplateStr_type, "_field_types",
TemplateStr_annotations) == 0;
if (!cond) {
Py_DECREF(TemplateStr_annotations);
return 0;
}
cond = PyObject_SetAttrString(state->TemplateStr_type, "__annotations__",
TemplateStr_annotations) == 0;
if (!cond) {
Py_DECREF(TemplateStr_annotations);
return 0;
}
Py_DECREF(TemplateStr_annotations);
PyObject *Constant_annotations = PyDict_New();
if (!Constant_annotations) return 0;
{
@ -6266,7 +6374,9 @@ init_types(void *arg)
" | Compare(expr left, cmpop* ops, expr* comparators)\n"
" | Call(expr func, expr* args, keyword* keywords)\n"
" | FormattedValue(expr value, int conversion, expr? format_spec)\n"
" | Interpolation(expr value, constant str, int conversion, expr? format_spec)\n"
" | JoinedStr(expr* values)\n"
" | TemplateStr(expr* values)\n"
" | Constant(constant value, string? kind)\n"
" | Attribute(expr value, identifier attr, expr_context ctx)\n"
" | Subscript(expr value, expr slice, expr_context ctx)\n"
@ -6361,10 +6471,22 @@ init_types(void *arg)
if (PyObject_SetAttr(state->FormattedValue_type, state->format_spec,
Py_None) == -1)
return -1;
state->Interpolation_type = make_type(state, "Interpolation",
state->expr_type,
Interpolation_fields, 4,
"Interpolation(expr value, constant str, int conversion, expr? format_spec)");
if (!state->Interpolation_type) return -1;
if (PyObject_SetAttr(state->Interpolation_type, state->format_spec,
Py_None) == -1)
return -1;
state->JoinedStr_type = make_type(state, "JoinedStr", state->expr_type,
JoinedStr_fields, 1,
"JoinedStr(expr* values)");
if (!state->JoinedStr_type) return -1;
state->TemplateStr_type = make_type(state, "TemplateStr", state->expr_type,
TemplateStr_fields, 1,
"TemplateStr(expr* values)");
if (!state->TemplateStr_type) return -1;
state->Constant_type = make_type(state, "Constant", state->expr_type,
Constant_fields, 2,
"Constant(constant value, string? kind)");
@ -8038,6 +8160,37 @@ _PyAST_FormattedValue(expr_ty value, int conversion, expr_ty format_spec, int
return p;
}
expr_ty
_PyAST_Interpolation(expr_ty value, constant str, int conversion, expr_ty
format_spec, int lineno, int col_offset, int end_lineno,
int end_col_offset, PyArena *arena)
{
expr_ty p;
if (!value) {
PyErr_SetString(PyExc_ValueError,
"field 'value' is required for Interpolation");
return NULL;
}
if (!str) {
PyErr_SetString(PyExc_ValueError,
"field 'str' is required for Interpolation");
return NULL;
}
p = (expr_ty)_PyArena_Malloc(arena, sizeof(*p));
if (!p)
return NULL;
p->kind = Interpolation_kind;
p->v.Interpolation.value = value;
p->v.Interpolation.str = str;
p->v.Interpolation.conversion = conversion;
p->v.Interpolation.format_spec = format_spec;
p->lineno = lineno;
p->col_offset = col_offset;
p->end_lineno = end_lineno;
p->end_col_offset = end_col_offset;
return p;
}
expr_ty
_PyAST_JoinedStr(asdl_expr_seq * values, int lineno, int col_offset, int
end_lineno, int end_col_offset, PyArena *arena)
@ -8055,6 +8208,23 @@ _PyAST_JoinedStr(asdl_expr_seq * values, int lineno, int col_offset, int
return p;
}
expr_ty
_PyAST_TemplateStr(asdl_expr_seq * values, int lineno, int col_offset, int
end_lineno, int end_col_offset, PyArena *arena)
{
expr_ty p;
p = (expr_ty)_PyArena_Malloc(arena, sizeof(*p));
if (!p)
return NULL;
p->kind = TemplateStr_kind;
p->v.TemplateStr.values = values;
p->lineno = lineno;
p->col_offset = col_offset;
p->end_lineno = end_lineno;
p->end_col_offset = end_col_offset;
return p;
}
expr_ty
_PyAST_Constant(constant value, string kind, int lineno, int col_offset, int
end_lineno, int end_col_offset, PyArena *arena)
@ -9674,6 +9844,31 @@ ast2obj_expr(struct ast_state *state, void* _o)
goto failed;
Py_DECREF(value);
break;
case Interpolation_kind:
tp = (PyTypeObject *)state->Interpolation_type;
result = PyType_GenericNew(tp, NULL, NULL);
if (!result) goto failed;
value = ast2obj_expr(state, o->v.Interpolation.value);
if (!value) goto failed;
if (PyObject_SetAttr(result, state->value, value) == -1)
goto failed;
Py_DECREF(value);
value = ast2obj_constant(state, o->v.Interpolation.str);
if (!value) goto failed;
if (PyObject_SetAttr(result, state->str, value) == -1)
goto failed;
Py_DECREF(value);
value = ast2obj_int(state, o->v.Interpolation.conversion);
if (!value) goto failed;
if (PyObject_SetAttr(result, state->conversion, value) == -1)
goto failed;
Py_DECREF(value);
value = ast2obj_expr(state, o->v.Interpolation.format_spec);
if (!value) goto failed;
if (PyObject_SetAttr(result, state->format_spec, value) == -1)
goto failed;
Py_DECREF(value);
break;
case JoinedStr_kind:
tp = (PyTypeObject *)state->JoinedStr_type;
result = PyType_GenericNew(tp, NULL, NULL);
@ -9685,6 +9880,17 @@ ast2obj_expr(struct ast_state *state, void* _o)
goto failed;
Py_DECREF(value);
break;
case TemplateStr_kind:
tp = (PyTypeObject *)state->TemplateStr_type;
result = PyType_GenericNew(tp, NULL, NULL);
if (!result) goto failed;
value = ast2obj_list(state, (asdl_seq*)o->v.TemplateStr.values,
ast2obj_expr);
if (!value) goto failed;
if (PyObject_SetAttr(result, state->values, value) == -1)
goto failed;
Py_DECREF(value);
break;
case Constant_kind:
tp = (PyTypeObject *)state->Constant_type;
result = PyType_GenericNew(tp, NULL, NULL);
@ -14793,6 +14999,91 @@ obj2ast_expr(struct ast_state *state, PyObject* obj, expr_ty* out, PyArena*
if (*out == NULL) goto failed;
return 0;
}
tp = state->Interpolation_type;
isinstance = PyObject_IsInstance(obj, tp);
if (isinstance == -1) {
return -1;
}
if (isinstance) {
expr_ty value;
constant str;
int conversion;
expr_ty format_spec;
if (PyObject_GetOptionalAttr(obj, state->value, &tmp) < 0) {
return -1;
}
if (tmp == NULL) {
PyErr_SetString(PyExc_TypeError, "required field \"value\" missing from Interpolation");
return -1;
}
else {
int res;
if (_Py_EnterRecursiveCall(" while traversing 'Interpolation' node")) {
goto failed;
}
res = obj2ast_expr(state, tmp, &value, arena);
_Py_LeaveRecursiveCall();
if (res != 0) goto failed;
Py_CLEAR(tmp);
}
if (PyObject_GetOptionalAttr(obj, state->str, &tmp) < 0) {
return -1;
}
if (tmp == NULL) {
PyErr_SetString(PyExc_TypeError, "required field \"str\" missing from Interpolation");
return -1;
}
else {
int res;
if (_Py_EnterRecursiveCall(" while traversing 'Interpolation' node")) {
goto failed;
}
res = obj2ast_constant(state, tmp, &str, arena);
_Py_LeaveRecursiveCall();
if (res != 0) goto failed;
Py_CLEAR(tmp);
}
if (PyObject_GetOptionalAttr(obj, state->conversion, &tmp) < 0) {
return -1;
}
if (tmp == NULL) {
PyErr_SetString(PyExc_TypeError, "required field \"conversion\" missing from Interpolation");
return -1;
}
else {
int res;
if (_Py_EnterRecursiveCall(" while traversing 'Interpolation' node")) {
goto failed;
}
res = obj2ast_int(state, tmp, &conversion, arena);
_Py_LeaveRecursiveCall();
if (res != 0) goto failed;
Py_CLEAR(tmp);
}
if (PyObject_GetOptionalAttr(obj, state->format_spec, &tmp) < 0) {
return -1;
}
if (tmp == NULL || tmp == Py_None) {
Py_CLEAR(tmp);
format_spec = NULL;
}
else {
int res;
if (_Py_EnterRecursiveCall(" while traversing 'Interpolation' node")) {
goto failed;
}
res = obj2ast_expr(state, tmp, &format_spec, arena);
_Py_LeaveRecursiveCall();
if (res != 0) goto failed;
Py_CLEAR(tmp);
}
*out = _PyAST_Interpolation(value, str, conversion, format_spec,
lineno, col_offset, end_lineno,
end_col_offset, arena);
if (*out == NULL) goto failed;
return 0;
}
tp = state->JoinedStr_type;
isinstance = PyObject_IsInstance(obj, tp);
if (isinstance == -1) {
@ -14844,6 +15135,57 @@ obj2ast_expr(struct ast_state *state, PyObject* obj, expr_ty* out, PyArena*
if (*out == NULL) goto failed;
return 0;
}
tp = state->TemplateStr_type;
isinstance = PyObject_IsInstance(obj, tp);
if (isinstance == -1) {
return -1;
}
if (isinstance) {
asdl_expr_seq* values;
if (PyObject_GetOptionalAttr(obj, state->values, &tmp) < 0) {
return -1;
}
if (tmp == NULL) {
tmp = PyList_New(0);
if (tmp == NULL) {
return -1;
}
}
{
int res;
Py_ssize_t len;
Py_ssize_t i;
if (!PyList_Check(tmp)) {
PyErr_Format(PyExc_TypeError, "TemplateStr field \"values\" must be a list, not a %.200s", _PyType_Name(Py_TYPE(tmp)));
goto failed;
}
len = PyList_GET_SIZE(tmp);
values = _Py_asdl_expr_seq_new(len, arena);
if (values == NULL) goto failed;
for (i = 0; i < len; i++) {
expr_ty val;
PyObject *tmp2 = Py_NewRef(PyList_GET_ITEM(tmp, i));
if (_Py_EnterRecursiveCall(" while traversing 'TemplateStr' node")) {
goto failed;
}
res = obj2ast_expr(state, tmp2, &val, arena);
_Py_LeaveRecursiveCall();
Py_DECREF(tmp2);
if (res != 0) goto failed;
if (len != PyList_GET_SIZE(tmp)) {
PyErr_SetString(PyExc_RuntimeError, "TemplateStr field \"values\" changed size during iteration");
goto failed;
}
asdl_seq_SET(values, i, val);
}
Py_CLEAR(tmp);
}
*out = _PyAST_TemplateStr(values, lineno, col_offset, end_lineno,
end_col_offset, arena);
if (*out == NULL) goto failed;
return 0;
}
tp = state->Constant_type;
isinstance = PyObject_IsInstance(obj, tp);
if (isinstance == -1) {
@ -17794,9 +18136,16 @@ astmodule_exec(PyObject *m)
< 0) {
return -1;
}
if (PyModule_AddObjectRef(m, "Interpolation", state->Interpolation_type) <
0) {
return -1;
}
if (PyModule_AddObjectRef(m, "JoinedStr", state->JoinedStr_type) < 0) {
return -1;
}
if (PyModule_AddObjectRef(m, "TemplateStr", state->TemplateStr_type) < 0) {
return -1;
}
if (PyModule_AddObjectRef(m, "Constant", state->Constant_type) < 0) {
return -1;
}

13
Python/ast.c
View File

@ -345,6 +345,9 @@ validate_expr(expr_ty exp, expr_context_ty ctx)
case JoinedStr_kind:
ret = validate_exprs(exp->v.JoinedStr.values, Load, 0);
break;
case TemplateStr_kind:
ret = validate_exprs(exp->v.TemplateStr.values, Load, 0);
break;
case FormattedValue_kind:
if (validate_expr(exp->v.FormattedValue.value, Load) == 0)
return 0;
@ -354,6 +357,15 @@ validate_expr(expr_ty exp, expr_context_ty ctx)
}
ret = 1;
break;
case Interpolation_kind:
if (validate_expr(exp->v.Interpolation.value, Load) == 0)
return 0;
if (exp->v.Interpolation.format_spec) {
ret = validate_expr(exp->v.Interpolation.format_spec, Load);
break;
}
ret = 1;
break;
case Attribute_kind:
ret = validate_expr(exp->v.Attribute.value, Load);
break;
@ -512,6 +524,7 @@ validate_pattern_match_value(expr_ty exp)
}
break;
case JoinedStr_kind:
case TemplateStr_kind:
// Handled in the later stages
return 1;
default:

7
Python/ast_opt.c
View File

@ -558,9 +558,16 @@ astfold_expr(expr_ty node_, PyArena *ctx_, _PyASTOptimizeState *state)
CALL(astfold_expr, expr_ty, node_->v.FormattedValue.value);
CALL_OPT(astfold_expr, expr_ty, node_->v.FormattedValue.format_spec);
break;
case Interpolation_kind:
CALL(astfold_expr, expr_ty, node_->v.Interpolation.value);
CALL_OPT(astfold_expr, expr_ty, node_->v.Interpolation.format_spec);
break;
case JoinedStr_kind:
CALL_SEQ(astfold_expr, expr, node_->v.JoinedStr.values);
break;
case TemplateStr_kind:
CALL_SEQ(astfold_expr, expr, node_->v.TemplateStr.values);
break;
case Attribute_kind:
CALL(astfold_expr, expr_ty, node_->v.Attribute.value);
break;

201
Python/ast_unparse.c
View File

@ -18,8 +18,12 @@ expr_as_unicode(expr_ty e, int level);
static int
append_ast_expr(PyUnicodeWriter *writer, expr_ty e, int level);
static int
append_templatestr(PyUnicodeWriter *writer, expr_ty e);
static int
append_joinedstr(PyUnicodeWriter *writer, expr_ty e, bool is_format_spec);
static int
append_interpolation(PyUnicodeWriter *writer, expr_ty e);
static int
append_formattedvalue(PyUnicodeWriter *writer, expr_ty e);
static int
append_ast_slice(PyUnicodeWriter *writer, expr_ty e);
@ -621,11 +625,15 @@ append_fstring_element(PyUnicodeWriter *writer, expr_ty e, bool is_format_spec)
return append_fstring_unicode(writer, e->v.Constant.value);
case JoinedStr_kind:
return append_joinedstr(writer, e, is_format_spec);
case TemplateStr_kind:
return append_templatestr(writer, e);
case FormattedValue_kind:
return append_formattedvalue(writer, e);
case Interpolation_kind:
return append_interpolation(writer, e);
default:
PyErr_SetString(PyExc_SystemError,
"unknown expression kind inside f-string");
"unknown expression kind inside f-string or t-string");
return -1;
}
}
@ -633,7 +641,7 @@ append_fstring_element(PyUnicodeWriter *writer, expr_ty e, bool is_format_spec)
/* Build body separately to enable wrapping the entire stream of Strs,
Constants and FormattedValues in one opening and one closing quote. */
static PyObject *
build_fstring_body(asdl_expr_seq *values, bool is_format_spec)
build_ftstring_body(asdl_expr_seq *values, bool is_format_spec)
{
PyUnicodeWriter *body_writer = PyUnicodeWriter_Create(256);
if (body_writer == NULL) {
@ -654,11 +662,99 @@ build_fstring_body(asdl_expr_seq *values, bool is_format_spec)
return PyUnicodeWriter_Finish(body_writer);
}
static int
_write_values_subarray(PyUnicodeWriter *writer, asdl_expr_seq *values, Py_ssize_t first_idx,
Py_ssize_t last_idx, char prefix, PyArena *arena)
{
int result = -1;
asdl_expr_seq *new_values = _Py_asdl_expr_seq_new(last_idx - first_idx + 1, arena);
if (!new_values) {
return result;
}
Py_ssize_t j = 0;
for (Py_ssize_t i = first_idx; i <= last_idx; ++i) {
asdl_seq_SET(new_values, j++, asdl_seq_GET(values, i));
}
PyObject *body = build_ftstring_body(new_values, false);
if (!body) {
return result;
}
if (-1 != append_char(writer, prefix) &&
-1 != append_repr(writer, body))
{
result = 0;
}
Py_DECREF(body);
return result;
}
static int
append_templatestr(PyUnicodeWriter *writer, expr_ty e)
{
PyArena *arena = _PyArena_New();
if (!arena) {
return -1;
}
Py_ssize_t last_idx = 0;
Py_ssize_t len = asdl_seq_LEN(e->v.TemplateStr.values);
for (Py_ssize_t i = 0; i < len; i++) {
expr_ty value = asdl_seq_GET(e->v.TemplateStr.values, i);
// Handle implicit concat of t-strings with f-strings
if (value->kind == FormattedValue_kind) {
if (i > last_idx) {
// Create a new TemplateStr with the values between last_idx and i
// and append it to the writer.
if (_write_values_subarray(writer, e->v.TemplateStr.values,
last_idx, i - 1, 't', arena) == -1) {
goto error;
}
if (append_charp(writer, " ") == -1) {
goto error;
}
}
// Append the FormattedValue to the writer.
if (_write_values_subarray(writer, e->v.TemplateStr.values,
i, i, 'f', arena) == -1) {
goto error;
}
if (i + 1 < len) {
if (append_charp(writer, " ") == -1) {
goto error;
}
}
last_idx = i + 1;
}
}
if (last_idx < len) {
if (_write_values_subarray(writer, e->v.TemplateStr.values,
last_idx, len - 1, 't', arena) == -1) {
goto error;
}
}
return 0;
error:
_PyArena_Free(arena);
return -1;
}
static int
append_joinedstr(PyUnicodeWriter *writer, expr_ty e, bool is_format_spec)
{
int result = -1;
PyObject *body = build_fstring_body(e->v.JoinedStr.values, is_format_spec);
PyObject *body = build_ftstring_body(e->v.JoinedStr.values, is_format_spec);
if (!body) {
return -1;
}
@ -678,13 +774,12 @@ append_joinedstr(PyUnicodeWriter *writer, expr_ty e, bool is_format_spec)
}
static int
append_formattedvalue(PyUnicodeWriter *writer, expr_ty e)
append_interpolation_value(PyUnicodeWriter *writer, expr_ty e)
{
const char *conversion;
const char *outer_brace = "{";
/* Grammar allows PR_TUPLE, but use >PR_TEST for adding parenthesis
around a lambda with ':' */
PyObject *temp_fv_str = expr_as_unicode(e->v.FormattedValue.value, PR_TEST + 1);
PyObject *temp_fv_str = expr_as_unicode(e, PR_TEST + 1);
if (!temp_fv_str) {
return -1;
}
@ -702,35 +797,81 @@ append_formattedvalue(PyUnicodeWriter *writer, expr_ty e)
return -1;
}
Py_DECREF(temp_fv_str);
return 0;
}
if (e->v.FormattedValue.conversion > 0) {
switch (e->v.FormattedValue.conversion) {
case 'a':
conversion = "!a";
break;
case 'r':
conversion = "!r";
break;
case 's':
conversion = "!s";
break;
default:
PyErr_SetString(PyExc_SystemError,
"unknown f-value conversion kind");
return -1;
}
APPEND_STR(conversion);
static int
append_interpolation_conversion(PyUnicodeWriter *writer, int conversion)
{
if (conversion < 0) {
return 0;
}
if (e->v.FormattedValue.format_spec) {
const char *conversion_str;
switch (conversion) {
case 'a':
conversion_str = "!a";
break;
case 'r':
conversion_str = "!r";
break;
case 's':
conversion_str = "!s";
break;
default:
PyErr_SetString(PyExc_SystemError,
"unknown f-value conversion kind");
return -1;
}
APPEND_STR(conversion_str);
return 0;
}
static int
append_interpolation_format_spec(PyUnicodeWriter *writer, expr_ty e)
{
if (e) {
if (-1 == PyUnicodeWriter_WriteChar(writer, ':') ||
-1 == append_fstring_element(writer,
e->v.FormattedValue.format_spec,
true
))
-1 == append_fstring_element(writer, e, true))
{
return -1;
}
}
return 0;
}
static int
append_interpolation(PyUnicodeWriter *writer, expr_ty e)
{
if (-1 == append_interpolation_value(writer, e->v.Interpolation.value)) {
return -1;
}
if (-1 == append_interpolation_conversion(writer, e->v.Interpolation.conversion)) {
return -1;
}
if (-1 == append_interpolation_format_spec(writer, e->v.Interpolation.format_spec)) {
return -1;
}
APPEND_STR_FINISH("}");
}
static int
append_formattedvalue(PyUnicodeWriter *writer, expr_ty e)
{
if (-1 == append_interpolation_value(writer, e->v.FormattedValue.value)) {
return -1;
}
if (-1 == append_interpolation_conversion(writer, e->v.FormattedValue.conversion)) {
return -1;
}
if (-1 == append_interpolation_format_spec(writer, e->v.FormattedValue.format_spec)) {
return -1;
}
APPEND_CHAR_FINISH('}');
}
@ -901,8 +1042,12 @@ append_ast_expr(PyUnicodeWriter *writer, expr_ty e, int level)
return append_ast_constant(writer, e->v.Constant.value);
case JoinedStr_kind:
return append_joinedstr(writer, e, false);
case TemplateStr_kind:
return append_templatestr(writer, e);
case FormattedValue_kind:
return append_formattedvalue(writer, e);
case Interpolation_kind:
return append_interpolation(writer, e);
/* The following exprs can be assignment targets. */
case Attribute_kind:
return append_ast_attribute(writer, e);

36
Python/bytecodes.c
View File

@ -16,6 +16,7 @@
#include "pycore_emscripten_signal.h" // _Py_CHECK_EMSCRIPTEN_SIGNALS
#include "pycore_function.h"
#include "pycore_instruments.h"
#include "pycore_interpolation.h" // _PyInterpolation_Build()
#include "pycore_intrinsics.h"
#include "pycore_long.h" // _PyLong_GetZero()
#include "pycore_moduleobject.h" // PyModuleObject
@ -30,6 +31,7 @@
#include "pycore_setobject.h" // _PySet_NextEntry()
#include "pycore_sliceobject.h" // _PyBuildSlice_ConsumeRefs
#include "pycore_stackref.h"
#include "pycore_template.h" // _PyTemplate_Build()
#include "pycore_tuple.h" // _PyTuple_ITEMS()
#include "pycore_typeobject.h" // _PySuper_Lookup()
@ -1903,6 +1905,40 @@ dummy_func(
str = PyStackRef_FromPyObjectSteal(str_o);
}
inst(BUILD_INTERPOLATION, (value, str, format[oparg & 1] -- interpolation)) {
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
PyObject *str_o = PyStackRef_AsPyObjectBorrow(str);
int conversion = oparg >> 2;
PyObject *format_o;
if (oparg & 1) {
format_o = PyStackRef_AsPyObjectBorrow(format[0]);
}
else {
format_o = &_Py_STR(empty);
}
PyObject *interpolation_o = _PyInterpolation_Build(value_o, str_o, conversion, format_o);
if (oparg & 1) {
PyStackRef_CLOSE(format[0]);
}
else {
DEAD(format);
}
PyStackRef_CLOSE(str);
PyStackRef_CLOSE(value);
ERROR_IF(interpolation_o == NULL);
interpolation = PyStackRef_FromPyObjectSteal(interpolation_o);
}
inst(BUILD_TEMPLATE, (strings, interpolations -- template)) {
PyObject *strings_o = PyStackRef_AsPyObjectBorrow(strings);
PyObject *interpolations_o = PyStackRef_AsPyObjectBorrow(interpolations);
PyObject *template_o = _PyTemplate_Build(strings_o, interpolations_o);
PyStackRef_CLOSE(interpolations);
PyStackRef_CLOSE(strings);
ERROR_IF(template_o == NULL);
template = PyStackRef_FromPyObjectSteal(template_o);
}
inst(BUILD_TUPLE, (values[oparg] -- tup)) {
PyObject *tup_o = _PyTuple_FromStackRefStealOnSuccess(values, oparg);
if (tup_o == NULL) {

2
Python/ceval.c
View File

@ -19,6 +19,7 @@
#include "pycore_import.h" // _PyImport_IsDefaultImportFunc()
#include "pycore_instruments.h"
#include "pycore_interpframe.h" // _PyFrame_SetStackPointer()
#include "pycore_interpolation.h" // _PyInterpolation_Build()
#include "pycore_intrinsics.h"
#include "pycore_jit.h"
#include "pycore_list.h" // _PyList_GetItemRef()
@ -36,6 +37,7 @@
#include "pycore_setobject.h" // _PySet_Update()
#include "pycore_sliceobject.h" // _PyBuildSlice_ConsumeRefs
#include "pycore_sysmodule.h" // _PySys_GetOptionalAttrString()
#include "pycore_template.h" // _PyTemplate_Build()
#include "pycore_traceback.h" // _PyTraceBack_FromFrame
#include "pycore_tuple.h" // _PyTuple_ITEMS()
#include "pycore_uop_ids.h" // Uops

112
Python/codegen.c
View File

@ -3607,7 +3607,9 @@ infer_type(expr_ty e)
case Lambda_kind:
return &PyFunction_Type;
case JoinedStr_kind:
case TemplateStr_kind:
case FormattedValue_kind:
case Interpolation_kind:
return &PyUnicode_Type;
case Constant_kind:
return Py_TYPE(e->v.Constant.value);
@ -3630,7 +3632,9 @@ check_caller(compiler *c, expr_ty e)
case SetComp_kind:
case GeneratorExp_kind:
case JoinedStr_kind:
case FormattedValue_kind: {
case TemplateStr_kind:
case FormattedValue_kind:
case Interpolation_kind: {
location loc = LOC(e);
return _PyCompile_Warn(c, loc, "'%.200s' object is not callable; "
"perhaps you missed a comma?",
@ -3693,7 +3697,9 @@ check_index(compiler *c, expr_ty e, expr_ty s)
case List_kind:
case ListComp_kind:
case JoinedStr_kind:
case FormattedValue_kind: {
case TemplateStr_kind:
case FormattedValue_kind:
case Interpolation_kind: {
location loc = LOC(e);
return _PyCompile_Warn(c, loc, "%.200s indices must be integers "
"or slices, not %.200s; "
@ -4043,6 +4049,59 @@ codegen_call(compiler *c, expr_ty e)
return ret;
}
static int
codegen_template_str(compiler *c, expr_ty e)
{
location loc = LOC(e);
expr_ty value;
Py_ssize_t value_count = asdl_seq_LEN(e->v.TemplateStr.values);
int last_was_interpolation = 1;
Py_ssize_t stringslen = 0;
for (Py_ssize_t i = 0; i < value_count; i++) {
value = asdl_seq_GET(e->v.TemplateStr.values, i);
if (value->kind == Interpolation_kind) {
if (last_was_interpolation) {
ADDOP_LOAD_CONST(c, loc, Py_NewRef(&_Py_STR(empty)));
stringslen++;
}
last_was_interpolation = 1;
}
else {
VISIT(c, expr, value);
Py_ssize_t j;
for (j = i + 1; j < value_count; j++) {
value = asdl_seq_GET(e->v.TemplateStr.values, j);
if (value->kind == Interpolation_kind) {
break;
}
VISIT(c, expr, value);
ADDOP_INPLACE(c, loc, Add);
}
i = j - 1;
stringslen++;
last_was_interpolation = 0;
}
}
if (last_was_interpolation) {
ADDOP_LOAD_CONST(c, loc, Py_NewRef(&_Py_STR(empty)));
stringslen++;
}
ADDOP_I(c, loc, BUILD_TUPLE, stringslen);
Py_ssize_t interpolationslen = 0;
for (Py_ssize_t i = 0; i < value_count; i++) {
value = asdl_seq_GET(e->v.TemplateStr.values, i);
if (value->kind == Interpolation_kind) {
VISIT(c, expr, value);
interpolationslen++;
}
}
ADDOP_I(c, loc, BUILD_TUPLE, interpolationslen);
ADDOP(c, loc, BUILD_TEMPLATE);
return SUCCESS;
}
static int
codegen_joined_str(compiler *c, expr_ty e)
{
@ -4072,24 +4131,41 @@ codegen_joined_str(compiler *c, expr_ty e)
return SUCCESS;
}
static int
codegen_interpolation(compiler *c, expr_ty e)
{
location loc = LOC(e);
VISIT(c, expr, e->v.Interpolation.value);
ADDOP_LOAD_CONST(c, loc, e->v.Interpolation.str);
int oparg = 2;
if (e->v.Interpolation.format_spec) {
oparg++;
VISIT(c, expr, e->v.Interpolation.format_spec);
}
int conversion = e->v.Interpolation.conversion;
if (conversion != -1) {
switch (conversion) {
case 's': oparg |= FVC_STR << 2; break;
case 'r': oparg |= FVC_REPR << 2; break;
case 'a': oparg |= FVC_ASCII << 2; break;
default:
PyErr_Format(PyExc_SystemError,
"Unrecognized conversion character %d", conversion);
return ERROR;
}
}
ADDOP_I(c, loc, BUILD_INTERPOLATION, oparg);
return SUCCESS;
}
/* Used to implement f-strings. Format a single value. */
static int
codegen_formatted_value(compiler *c, expr_ty e)
{
/* Our oparg encodes 2 pieces of information: the conversion
character, and whether or not a format_spec was provided.
Convert the conversion char to 3 bits:
: 000 0x0 FVC_NONE The default if nothing specified.
!s : 001 0x1 FVC_STR
!r : 010 0x2 FVC_REPR
!a : 011 0x3 FVC_ASCII
next bit is whether or not we have a format spec:
yes : 100 0x4
no : 000 0x0
*/
int conversion = e->v.FormattedValue.conversion;
int oparg;
@ -5182,8 +5258,12 @@ codegen_visit_expr(compiler *c, expr_ty e)
break;
case JoinedStr_kind:
return codegen_joined_str(c, e);
case TemplateStr_kind:
return codegen_template_str(c, e);
case FormattedValue_kind:
return codegen_formatted_value(c, e);
case Interpolation_kind:
return codegen_interpolation(c, e);
/* The following exprs can be assignment targets. */
case Attribute_kind:
if (e->v.Attribute.ctx == Load) {

83
Python/executor_cases.c.h generated
View File

@ -2604,6 +2604,89 @@
break;
}
case _BUILD_INTERPOLATION: {
_PyStackRef *format;
_PyStackRef str;
_PyStackRef value;
_PyStackRef interpolation;
oparg = CURRENT_OPARG();
format = &stack_pointer[-(oparg & 1)];
str = stack_pointer[-1 - (oparg & 1)];
value = stack_pointer[-2 - (oparg & 1)];
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
PyObject *str_o = PyStackRef_AsPyObjectBorrow(str);
int conversion = oparg >> 2;
PyObject *format_o;
if (oparg & 1) {
format_o = PyStackRef_AsPyObjectBorrow(format[0]);
}
else {
format_o = &_Py_STR(empty);
}
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *interpolation_o = _PyInterpolation_Build(value_o, str_o, conversion, format_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (oparg & 1) {
stack_pointer += -(oparg & 1);
assert(WITHIN_STACK_BOUNDS());
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(format[0]);
stack_pointer = _PyFrame_GetStackPointer(frame);
}
else {
stack_pointer += -(oparg & 1);
}
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(str);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (interpolation_o == NULL) {
JUMP_TO_ERROR();
}
interpolation = PyStackRef_FromPyObjectSteal(interpolation_o);
stack_pointer[0] = interpolation;
stack_pointer += 1;
assert(WITHIN_STACK_BOUNDS());
break;
}
case _BUILD_TEMPLATE: {
_PyStackRef interpolations;
_PyStackRef strings;
_PyStackRef template;
interpolations = stack_pointer[-1];
strings = stack_pointer[-2];
PyObject *strings_o = PyStackRef_AsPyObjectBorrow(strings);
PyObject *interpolations_o = PyStackRef_AsPyObjectBorrow(interpolations);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *template_o = _PyTemplate_Build(strings_o, interpolations_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(interpolations);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(strings);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (template_o == NULL) {
JUMP_TO_ERROR();
}
template = PyStackRef_FromPyObjectSteal(template_o);
stack_pointer[0] = template;
stack_pointer += 1;
assert(WITHIN_STACK_BOUNDS());
break;
}
case _BUILD_TUPLE: {
_PyStackRef *values;
_PyStackRef tup;

96
Python/generated_cases.c.h generated
View File

@ -1088,6 +1088,64 @@
DISPATCH();
}
TARGET(BUILD_INTERPOLATION) {
#if Py_TAIL_CALL_INTERP
int opcode = BUILD_INTERPOLATION;
(void)(opcode);
#endif
frame->instr_ptr = next_instr;
next_instr += 1;
INSTRUCTION_STATS(BUILD_INTERPOLATION);
_PyStackRef value;
_PyStackRef str;
_PyStackRef *format;
_PyStackRef interpolation;
format = &stack_pointer[-(oparg & 1)];
str = stack_pointer[-1 - (oparg & 1)];
value = stack_pointer[-2 - (oparg & 1)];
PyObject *value_o = PyStackRef_AsPyObjectBorrow(value);
PyObject *str_o = PyStackRef_AsPyObjectBorrow(str);
int conversion = oparg >> 2;
PyObject *format_o;
if (oparg & 1) {
format_o = PyStackRef_AsPyObjectBorrow(format[0]);
}
else {
format_o = &_Py_STR(empty);
}
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *interpolation_o = _PyInterpolation_Build(value_o, str_o, conversion, format_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (oparg & 1) {
stack_pointer += -(oparg & 1);
assert(WITHIN_STACK_BOUNDS());
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(format[0]);
stack_pointer = _PyFrame_GetStackPointer(frame);
}
else {
stack_pointer += -(oparg & 1);
}
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(str);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(value);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (interpolation_o == NULL) {
JUMP_TO_LABEL(error);
}
interpolation = PyStackRef_FromPyObjectSteal(interpolation_o);
stack_pointer[0] = interpolation;
stack_pointer += 1;
assert(WITHIN_STACK_BOUNDS());
DISPATCH();
}
TARGET(BUILD_LIST) {
#if Py_TAIL_CALL_INTERP
int opcode = BUILD_LIST;
@ -1303,6 +1361,44 @@
DISPATCH();
}
TARGET(BUILD_TEMPLATE) {
#if Py_TAIL_CALL_INTERP
int opcode = BUILD_TEMPLATE;
(void)(opcode);
#endif
frame->instr_ptr = next_instr;
next_instr += 1;
INSTRUCTION_STATS(BUILD_TEMPLATE);
_PyStackRef strings;
_PyStackRef interpolations;
_PyStackRef template;
interpolations = stack_pointer[-1];
strings = stack_pointer[-2];
PyObject *strings_o = PyStackRef_AsPyObjectBorrow(strings);
PyObject *interpolations_o = PyStackRef_AsPyObjectBorrow(interpolations);
_PyFrame_SetStackPointer(frame, stack_pointer);
PyObject *template_o = _PyTemplate_Build(strings_o, interpolations_o);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(interpolations);
stack_pointer = _PyFrame_GetStackPointer(frame);
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
_PyFrame_SetStackPointer(frame, stack_pointer);
PyStackRef_CLOSE(strings);
stack_pointer = _PyFrame_GetStackPointer(frame);
if (template_o == NULL) {
JUMP_TO_LABEL(error);
}
template = PyStackRef_FromPyObjectSteal(template_o);
stack_pointer[0] = template;
stack_pointer += 1;
assert(WITHIN_STACK_BOUNDS());
DISPATCH();
}
TARGET(BUILD_TUPLE) {
#if Py_TAIL_CALL_INTERP
int opcode = BUILD_TUPLE;

2
Python/jit.c
View File

@ -12,6 +12,7 @@
#include "pycore_frame.h"
#include "pycore_function.h"
#include "pycore_interpframe.h"
#include "pycore_interpolation.h"
#include "pycore_intrinsics.h"
#include "pycore_list.h"
#include "pycore_long.h"
@ -21,6 +22,7 @@
#include "pycore_pyerrors.h"
#include "pycore_setobject.h"
#include "pycore_sliceobject.h"
#include "pycore_template.h"
#include "pycore_tuple.h"
#include "pycore_unicodeobject.h"

14
Python/opcode_targets.h generated
View File

@ -2,8 +2,9 @@
static void *opcode_targets[256] = {
&&TARGET_CACHE,
&&TARGET_BINARY_SLICE,
&&TARGET_CALL_FUNCTION_EX,
&&TARGET_BUILD_TEMPLATE,
&&TARGET_BINARY_OP_INPLACE_ADD_UNICODE,
&&TARGET_CALL_FUNCTION_EX,
&&TARGET_CHECK_EG_MATCH,
&&TARGET_CHECK_EXC_MATCH,
&&TARGET_CLEANUP_THROW,
@ -16,8 +17,8 @@ static void *opcode_targets[256] = {
&&TARGET_GET_AITER,
&&TARGET_GET_ANEXT,
&&TARGET_GET_ITER,
&&TARGET_GET_LEN,
&&TARGET_RESERVED,
&&TARGET_GET_LEN,
&&TARGET_GET_YIELD_FROM_ITER,
&&TARGET_INTERPRETER_EXIT,
&&TARGET_LOAD_BUILD_CLASS,
@ -44,6 +45,7 @@ static void *opcode_targets[256] = {
&&TARGET_UNARY_NOT,
&&TARGET_WITH_EXCEPT_START,
&&TARGET_BINARY_OP,
&&TARGET_BUILD_INTERPOLATION,
&&TARGET_BUILD_LIST,
&&TARGET_BUILD_MAP,
&&TARGET_BUILD_SET,
@ -126,8 +128,6 @@ static void *opcode_targets[256] = {
&&_unknown_opcode,
&&_unknown_opcode,
&&_unknown_opcode,
&&_unknown_opcode,
&&_unknown_opcode,
&&TARGET_RESUME,
&&TARGET_BINARY_OP_ADD_FLOAT,
&&TARGET_BINARY_OP_ADD_INT,
@ -285,11 +285,13 @@ Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BINARY_OP_SUBSCR_TUPLE_INT(TAIL_
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BINARY_OP_SUBTRACT_FLOAT(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BINARY_OP_SUBTRACT_INT(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BINARY_SLICE(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BUILD_INTERPOLATION(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BUILD_LIST(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BUILD_MAP(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BUILD_SET(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BUILD_SLICE(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BUILD_STRING(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BUILD_TEMPLATE(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_BUILD_TUPLE(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_CACHE(TAIL_CALL_PARAMS);
Py_PRESERVE_NONE_CC static PyObject *_TAIL_CALL_CALL(TAIL_CALL_PARAMS);
@ -521,11 +523,13 @@ static py_tail_call_funcptr INSTRUCTION_TABLE[256] = {
[BINARY_OP_SUBTRACT_FLOAT] = _TAIL_CALL_BINARY_OP_SUBTRACT_FLOAT,
[BINARY_OP_SUBTRACT_INT] = _TAIL_CALL_BINARY_OP_SUBTRACT_INT,
[BINARY_SLICE] = _TAIL_CALL_BINARY_SLICE,
[BUILD_INTERPOLATION] = _TAIL_CALL_BUILD_INTERPOLATION,
[BUILD_LIST] = _TAIL_CALL_BUILD_LIST,
[BUILD_MAP] = _TAIL_CALL_BUILD_MAP,
[BUILD_SET] = _TAIL_CALL_BUILD_SET,
[BUILD_SLICE] = _TAIL_CALL_BUILD_SLICE,
[BUILD_STRING] = _TAIL_CALL_BUILD_STRING,
[BUILD_TEMPLATE] = _TAIL_CALL_BUILD_TEMPLATE,
[BUILD_TUPLE] = _TAIL_CALL_BUILD_TUPLE,
[CACHE] = _TAIL_CALL_CACHE,
[CALL] = _TAIL_CALL_CALL,
@ -729,8 +733,6 @@ static py_tail_call_funcptr INSTRUCTION_TABLE[256] = {
[UNPACK_SEQUENCE_TWO_TUPLE] = _TAIL_CALL_UNPACK_SEQUENCE_TWO_TUPLE,
[WITH_EXCEPT_START] = _TAIL_CALL_WITH_EXCEPT_START,
[YIELD_VALUE] = _TAIL_CALL_YIELD_VALUE,
[119] = _TAIL_CALL_UNKNOWN_OPCODE,
[120] = _TAIL_CALL_UNKNOWN_OPCODE,
[121] = _TAIL_CALL_UNKNOWN_OPCODE,
[122] = _TAIL_CALL_UNKNOWN_OPCODE,
[123] = _TAIL_CALL_UNKNOWN_OPCODE,

18
Python/optimizer_cases.c.h generated
View File

@ -1039,6 +1039,24 @@
break;
}
case _BUILD_INTERPOLATION: {
JitOptSymbol *interpolation;
interpolation = sym_new_not_null(ctx);
stack_pointer[-2 - (oparg & 1)] = interpolation;
stack_pointer += -1 - (oparg & 1);
assert(WITHIN_STACK_BOUNDS());
break;
}
case _BUILD_TEMPLATE: {
JitOptSymbol *template;
template = sym_new_not_null(ctx);
stack_pointer[-2] = template;
stack_pointer += -1;
assert(WITHIN_STACK_BOUNDS());
break;
}
case _BUILD_TUPLE: {
JitOptSymbol **values;
JitOptSymbol *tup;

6
Python/pylifecycle.c
View File

@ -13,6 +13,7 @@
#include "pycore_freelist.h" // _PyObject_ClearFreeLists()
#include "pycore_global_objects_fini_generated.h" // _PyStaticObjects_CheckRefcnt()
#include "pycore_initconfig.h" // _PyStatus_OK()
#include "pycore_interpolation.h" // _PyInterpolation_InitTypes()
#include "pycore_long.h" // _PyLong_InitTypes()
#include "pycore_object.h" // _PyDebug_PrintTotalRefs()
#include "pycore_obmalloc.h" // _PyMem_init_obmalloc()
@ -754,6 +755,11 @@ pycore_init_types(PyInterpreterState *interp)
return status;
}
status = _PyInterpolation_InitTypes(interp);
if (_PyStatus_EXCEPTION(status)) {
return status;
}
return _PyStatus_OK();
}

8
Python/symtable.c
View File

@ -2510,9 +2510,17 @@ symtable_visit_expr(struct symtable *st, expr_ty e)
if (e->v.FormattedValue.format_spec)
VISIT(st, expr, e->v.FormattedValue.format_spec);
break;
case Interpolation_kind:
VISIT(st, expr, e->v.Interpolation.value);
if (e->v.Interpolation.format_spec)
VISIT(st, expr, e->v.Interpolation.format_spec);
break;
case JoinedStr_kind:
VISIT_SEQ(st, expr, e->v.JoinedStr.values);
break;
case TemplateStr_kind:
VISIT_SEQ(st, expr, e->v.TemplateStr.values);
break;
case Constant_kind:
/* Nothing to do here. */
break;

3
Tools/build/generate_token.py
View File

@ -87,7 +87,8 @@ extern "C" {
(x) == INDENT || \\
(x) == DEDENT)
#define ISSTRINGLIT(x) ((x) == STRING || \\
(x) == FSTRING_MIDDLE)
(x) == FSTRING_MIDDLE || \\
(x) == TSTRING_MIDDLE)
// Export these 4 symbols for 'test_peg_generator'

3
Tools/c-analyzer/TODO
View File

@ -794,6 +794,7 @@ Objects/genobject.c:_PyAsyncGenASend_Type PyTypeObject _P
Objects/genobject.c:_PyAsyncGenAThrow_Type PyTypeObject _PyAsyncGenAThrow_Type
Objects/genobject.c:_PyAsyncGenWrappedValue_Type PyTypeObject _PyAsyncGenWrappedValue_Type
Objects/genobject.c:_PyCoroWrapper_Type PyTypeObject _PyCoroWrapper_Type
Objects/interpolationobject.c:_PyInterpolation_Type PyTypeObject _PyInterpolation_Type
Objects/interpreteridobject.c:_PyInterpreterID_Type PyTypeObject _PyInterpreterID_Type
Objects/iterobject.c:PyCallIter_Type PyTypeObject PyCallIter_Type
Objects/iterobject.c:PySeqIter_Type PyTypeObject PySeqIter_Type
@ -827,6 +828,8 @@ Objects/sliceobject.c:PyEllipsis_Type PyTypeObject Py
Objects/sliceobject.c:PySlice_Type PyTypeObject PySlice_Type
Objects/stringlib/unicode_format.h:PyFieldNameIter_Type static PyTypeObject PyFieldNameIter_Type
Objects/stringlib/unicode_format.h:PyFormatterIter_Type static PyTypeObject PyFormatterIter_Type
Objects/templateobject.c:_PyTemplateIter_Type PyTypeObject _PyTemplateIter_Type
Objects/templateobject.c:_PyTemplate_Type PyTypeObject _PyTemplate_Type
Objects/tupleobject.c:PyTupleIter_Type PyTypeObject PyTupleIter_Type
Objects/tupleobject.c:PyTuple_Type PyTypeObject PyTuple_Type
Objects/typeobject.c:PyBaseObject_Type PyTypeObject PyBaseObject_Type

3
Tools/c-analyzer/cpython/globals-to-fix.tsv
View File

@ -55,6 +55,7 @@ Objects/genobject.c - _PyAsyncGenASend_Type -
Objects/genobject.c - _PyAsyncGenAThrow_Type -
Objects/genobject.c - _PyAsyncGenWrappedValue_Type -
Objects/genobject.c - _PyCoroWrapper_Type -
Objects/interpolationobject.c - _PyInterpolation_Type -
Objects/iterobject.c - PyCallIter_Type -
Objects/iterobject.c - PySeqIter_Type -
Objects/iterobject.c - _PyAnextAwaitable_Type -
@ -86,6 +87,8 @@ Objects/setobject.c - PySetIter_Type -
Objects/setobject.c - PySet_Type -
Objects/sliceobject.c - PyEllipsis_Type -
Objects/sliceobject.c - PySlice_Type -
Objects/templateobject.c - _PyTemplateIter_Type -
Objects/templateobject.c - _PyTemplate_Type -
Objects/tupleobject.c - PyTupleIter_Type -
Objects/tupleobject.c - PyTuple_Type -
Objects/typeobject.c - _PyBufferWrapper_Type -

Can't render this file because it has a wrong number of fields in line 4.

2
Tools/jit/template.c
View File

@ -13,6 +13,7 @@
#include "pycore_function.h"
#include "pycore_genobject.h"
#include "pycore_interpframe.h"
#include "pycore_interpolation.h"
#include "pycore_intrinsics.h"
#include "pycore_jit.h"
#include "pycore_list.h"
@ -25,6 +26,7 @@
#include "pycore_setobject.h"
#include "pycore_sliceobject.h"
#include "pycore_stackref.h"
#include "pycore_template.h"
#include "pycore_tuple.h"
#include "pycore_unicodeobject.h"