102 Commits

Author SHA1 Message Date
Antoine Pitrou
5a24d82941 Add a test for issue #1813: getlocale() failing under a Turkish locale
(not a problem under 3.x)
2011-07-24 02:41:54 +02:00
Antoine Pitrou
cf9d3c08c8 Issue #1813: Fix codec lookup under Turkish locales. 2011-07-24 02:27:04 +02:00
Victor Stinner
0501070669 Revert my commit 3555cf6f9c98: "Issue #8796: codecs.open() calls the builtin
open() function instead of using StreamReaderWriter. Deprecate StreamReader,
StreamWriter, StreamReaderWriter, StreamRecoder and EncodedFile() of the codec
module. Use the builtin open() function or io.TextIOWrapper instead."

"It has not been approved !" wrote Marc-Andre Lemburg.
2011-05-27 16:50:40 +02:00
Victor Stinner
98fe1a0c3b Issue #8796: codecs.open() calls the builtin open() function instead of using
StreamReaderWriter. Deprecate StreamReader, StreamWriter, StreamReaderWriter,
StreamRecoder and EncodedFile() of the codec module. Use the builtin open()
function or io.TextIOWrapper instead.
2011-05-27 01:51:18 +02:00
Victor Stinner
d6881701fb Merge 3.2 2011-05-23 14:58:07 +02:00
Victor Stinner
b43dd4b8ca Merge 3.1 2011-05-23 14:57:05 +02:00
Victor Stinner
2cca057284 test_codecs now removes the temporay file (created by the test) 2011-05-23 14:51:42 +02:00
Marc-André Lemburg
8f36af7a4c Normalize the encoding names for Latin-1 and UTF-8 to
'latin-1' and 'utf-8'.

These are optimized in the Python Unicode implementation
to result in more direct processing, bypassing the codec
registry.

Also see issue11303.
2011-02-25 15:42:01 +00:00
Benjamin Peterson
28a4dce6a8 remove (un)transform methods 2010-12-12 01:33:04 +00:00
Victor Stinner
53a9dd776e Issue #10546: UTF-16-LE and UTF-16-BE *do* support non-BMP characters
Fix the doc and add tests.
2010-12-08 22:25:45 +00:00
Georg Brandl
02524629f3 #7475: add (un)transform method to bytes/bytearray and str, add back codecs that can be used with them from Python 2. 2010-12-02 18:06:51 +00:00
Ezio Melotti
19f2aeba67 Merged revisions 86596 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r86596 | ezio.melotti | 2010-11-20 21:04:17 +0200 (Sat, 20 Nov 2010) | 1 line

  #9424: Replace deprecated assert* methods in the Python test suite.
........
2010-11-21 01:30:29 +00:00
Ezio Melotti
b3aedd4862 #9424: Replace deprecated assert* methods in the Python test suite. 2010-11-20 19:04:17 +00:00
Benjamin Peterson
5a6214afe2 Merged revisions 81499,81506 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r81499 | georg.brandl | 2010-05-24 16:29:07 -0500 (Mon, 24 May 2010) | 1 line

  #8016: add the CP858 codec (approved by Benjamin).  (Also add CP720 to the tests, it was missing there.)
........
  r81506 | benjamin.peterson | 2010-05-24 17:04:53 -0500 (Mon, 24 May 2010) | 1 line

  set svn:eol-style
........
2010-06-27 22:41:29 +00:00
Victor Stinner
554f3f0081 Issue #850997: mbcs encoding (Windows only) handles errors argument: strict
mode raises unicode errors. The encoder only supports "strict" and "replace"
error handlers, the decoder only supports "strict" and "ignore" error handlers.
2010-06-16 23:33:54 +00:00
Antoine Pitrou
6107a688ee Merged revisions 81908 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
  r81908 | antoine.pitrou | 2010-06-11 23:46:32 +0200 (ven., 11 juin 2010) | 11 lines

  Merged revisions 81907 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ........
    r81907 | antoine.pitrou | 2010-06-11 23:42:26 +0200 (ven., 11 juin 2010) | 5 lines

    Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash
    the interpreter with characters outside the Basic Multilingual Plane
    (higher than 0x10000).
  ........
................
2010-06-11 21:48:34 +00:00
Antoine Pitrou
cc0cfd3576 Merged revisions 81907 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r81907 | antoine.pitrou | 2010-06-11 23:42:26 +0200 (ven., 11 juin 2010) | 5 lines

  Issue #8941: decoding big endian UTF-32 data in UCS-2 builds could crash
  the interpreter with characters outside the Basic Multilingual Plane
  (higher than 0x10000).
........
2010-06-11 21:46:32 +00:00
Philip Jenvey
ddf0d0383c Merged revisions 79780 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
  r79780 | philip.jenvey | 2010-04-04 20:05:24 -0700 (Sun, 04 Apr 2010) | 9 lines

  Merged revisions 79779 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ........
    r79779 | philip.jenvey | 2010-04-04 19:51:51 -0700 (Sun, 04 Apr 2010) | 2 lines

    fix escape_encode to return the correct consumed size
  ........
................
2010-06-09 17:56:11 +00:00
Victor Stinner
3dcb5acdb0 Issue #8838, #8339: Remove codecs.charbuffer_encode() and "t#" parsing format
Remove last references to the "char buffer" of the buffer protocol from
Python3.
2010-06-08 22:54:19 +00:00
Victor Stinner
b64d0eba50 Merged revisions 81474 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
  r81474 | victor.stinner | 2010-05-22 18:59:09 +0200 (sam., 22 mai 2010) | 20 lines

  Merged revisions 81471-81472 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ........
    r81471 | victor.stinner | 2010-05-22 15:37:56 +0200 (sam., 22 mai 2010) | 7 lines

    Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32

     * Fix seek() method of codecs.open(), don't write the BOM twice after seek(0)
     * Fix reset() method of codecs, UTF-16, UTF-32 and StreamWriter classes
     * test_codecs: use "w+" mode instead of "wt+". "t" mode is not supported by
       Solaris or Windows, but does it really exist? I found it the in the issue.
  ........
    r81472 | victor.stinner | 2010-05-22 15:44:25 +0200 (sam., 22 mai 2010) | 4 lines

    Fix my last commit (r81471) about codecs

    Rememder: don't touch the code just before a commit
  ........
................
2010-05-22 17:01:13 +00:00
Victor Stinner
a92ad7ee2c Merged revisions 81471-81472 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r81471 | victor.stinner | 2010-05-22 15:37:56 +0200 (sam., 22 mai 2010) | 7 lines

  Issue #6268: More bugfixes about BOM, UTF-16 and UTF-32

   * Fix seek() method of codecs.open(), don't write the BOM twice after seek(0)
   * Fix reset() method of codecs, UTF-16, UTF-32 and StreamWriter classes
   * test_codecs: use "w+" mode instead of "wt+". "t" mode is not supported by
     Solaris or Windows, but does it really exist? I found it the in the issue.
........
  r81472 | victor.stinner | 2010-05-22 15:44:25 +0200 (sam., 22 mai 2010) | 4 lines

  Fix my last commit (r81471) about codecs

  Rememder: don't touch the code just before a commit
........
2010-05-22 16:59:09 +00:00
Victor Stinner
37b8200608 Merged revisions 81461 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
  r81461 | victor.stinner | 2010-05-22 04:16:27 +0200 (sam., 22 mai 2010) | 10 lines

  Merged revisions 81459 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ........
    r81459 | victor.stinner | 2010-05-22 04:11:07 +0200 (sam., 22 mai 2010) | 3 lines

    Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice
    after seek(0)
  ........
................
2010-05-22 02:17:42 +00:00
Victor Stinner
3fed0870a6 Merged revisions 81459 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r81459 | victor.stinner | 2010-05-22 04:11:07 +0200 (sam., 22 mai 2010) | 3 lines

  Issue #6268: Fix seek() method of codecs.open(), don't read the BOM twice
  after seek(0)
........
2010-05-22 02:16:27 +00:00
Victor Stinner
158701d886 Merged revisions 80382 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r80382 | victor.stinner | 2010-04-22 21:38:16 +0200 (jeu., 22 avril 2010) | 3 lines

  Issue #8092: Fix PyUnicode_EncodeUTF8() to support error handler producing
  unicode string (eg. backslashreplace)
........
2010-04-22 19:41:01 +00:00
Victor Stinner
31be90b0c7 Issue #8092: Fix PyUnicode_EncodeUTF8() to support error handler producing
unicode string (eg. backslashreplace)
2010-04-22 19:38:16 +00:00
Philip Jenvey
66a1bd5568 Merged revisions 79779 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r79779 | philip.jenvey | 2010-04-04 19:51:51 -0700 (Sun, 04 Apr 2010) | 2 lines

  fix escape_encode to return the correct consumed size
........
2010-04-05 03:05:24 +00:00
Florent Xicluna
e36b2c693c Recorded merge of revisions 78462,78484 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

................
  r78462 | florent.xicluna | 2010-02-26 12:12:33 +0100 (ven, 26 fév 2010) | 9 lines

  Merged revisions 78461 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ........
    r78461 | florent.xicluna | 2010-02-26 11:40:58 +0100 (ven, 26 fév 2010) | 2 lines

    #691291: codecs.open() should not convert end of lines on reading and writing.
  ........
................
  r78484 | florent.xicluna | 2010-02-27 12:31:21 +0100 (sam, 27 fév 2010) | 9 lines

  Merged revisions 78482 via svnmerge from
  svn+ssh://pythondev@svn.python.org/python/trunk

  ........
    r78482 | florent.xicluna | 2010-02-27 12:19:18 +0100 (sam, 27 fév 2010) | 2 lines

    Add entry for issue #691291.
  ........
................
2010-02-27 11:38:27 +00:00
Florent Xicluna
c1c415f304 Merged revisions 78461 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r78461 | florent.xicluna | 2010-02-26 11:40:58 +0100 (ven, 26 fév 2010) | 2 lines

  #691291: codecs.open() should not convert end of lines on reading and writing.
........
2010-02-26 11:12:33 +00:00
Ezio Melotti
e96159335f Merged revisions 77727 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r77727 | ezio.melotti | 2010-01-24 18:58:36 +0200 (Sun, 24 Jan 2010) | 1 line

  use assert[Not]IsInstance where appropriate
........
2010-01-24 19:26:24 +00:00
Georg Brandl
7b10c9f301 Merged revisions 74871 via svnmerge from
svn+ssh://svn.python.org/python/branches/py3k

................
  r74871 | georg.brandl | 2009-09-17 13:41:24 +0200 (Do, 17 Sep 2009) | 12 lines

  Merged revisions 74869 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk

  (Only the new tests, the code had already been corrected due to an API change
  in unicode_decode_call_errorhandler.)

  ........
    r74869 | georg.brandl | 2009-09-17 13:28:09 +0200 (Do, 17 Sep 2009) | 4 lines

    Issue #6922: Fix an infinite loop when trying to decode an invalid
    UTF-32 stream with a non-raising error handler like "replace" or "ignore".
  ........
................
2009-09-17 11:46:23 +00:00
Georg Brandl
791f4e15db Merged revisions 74869 via svnmerge from svn+ssh://pythondev@svn.python.org/python/trunk
(Only the new tests, the code had already been corrected due to an API change
in unicode_decode_call_errorhandler.)

........
  r74869 | georg.brandl | 2009-09-17 13:28:09 +0200 (Do, 17 Sep 2009) | 4 lines

  Issue #6922: Fix an infinite loop when trying to decode an invalid
  UTF-32 stream with a non-raising error handler like "replace" or "ignore".
........
2009-09-17 11:41:24 +00:00
Georg Brandl
ab91fdef1f Merged revisions 73715 via svnmerge from
svn+ssh://svn.python.org/python/branches/py3k

........
  r73715 | benjamin.peterson | 2009-07-01 01:06:06 +0200 (Mi, 01 Jul 2009) | 1 line

  convert old fail* assertions to assert*
........
2009-08-13 08:51:18 +00:00
Benjamin Peterson
c9c0f201fe convert old fail* assertions to assert* 2009-06-30 23:06:06 +00:00
Amaury Forgeot d'Arc
e5344d6c45 Merged revisions 73698 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/branches/py3k

........
  r73698 | amaury.forgeotdarc | 2009-06-30 00:36:49 +0200 (mar., 30 juin 2009) | 7 lines

  #6373: SystemError in str.encode('latin1', 'surrogateescape')
  if the string contains unpaired surrogates.
  (In debug build, crash in assert())

  This can happen with normal processing, if python starts with utf-8,
  then calls sys.setfilesystemencoding('latin-1')
........
2009-06-29 22:38:54 +00:00
Amaury Forgeot d'Arc
84ec8d9314 #6373: SystemError in str.encode('latin1', 'surrogateescape')
if the string contains unpaired surrogates.
(In debug build, crash in assert())

This can happen with normal processing, if python starts with utf-8,
then calls sys.setfilesystemencoding('latin-1')
2009-06-29 22:36:49 +00:00
Martin v. Löwis
43c57785d3 Rename utf8b error handler to surrogateescape. 2009-05-10 08:15:24 +00:00
Martin v. Löwis
e0a2b72e61 Rename the surrogates error handler to surrogatepass. 2009-05-10 08:08:56 +00:00
Walter Dörwald
8dc33d56f5 Merged revisions 72404-72406 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r72404 | walter.doerwald | 2009-05-06 16:28:24 +0200 (Mi, 06 Mai 2009) | 3 lines

  Issue 3739: The unicode-internal encoder now reports the number of *characters*
  consumed like any other encoder (instead of the number of bytes).
........
  r72406 | walter.doerwald | 2009-05-06 16:32:35 +0200 (Mi, 06 Mai 2009) | 2 lines

  Add NEWS entry about issue #3739.
........
2009-05-06 14:41:26 +00:00
Martin v. Löwis
011e842033 Issue #5915: Implement PEP 383, Non-decodable Bytes in
System Character Interfaces.
2009-05-05 04:43:17 +00:00
Martin v. Löwis
db12d454e6 Issue #3672: Reject surrogates in utf-8 codec; add surrogates error
handler.
2009-05-02 18:52:14 +00:00
Antoine Pitrou
81fabdb437 Issue #4874: Most builtin decoders now reject unicode input. 2009-01-22 10:11:36 +00:00
Antoine Pitrou
616d28566b Issue #2394: implement more of the memoryview API. 2008-08-19 22:09:34 +00:00
Benjamin Peterson
ee8712cda4 #2621 rename test.test_support to test.support 2008-05-20 21:35:26 +00:00
Christian Heimes
5d14c2b8f8 Merged revisions 59056-59076 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r59064 | christian.heimes | 2007-11-20 02:48:48 +0100 (Tue, 20 Nov 2007) | 1 line

  Fixed bug #1470
........
  r59066 | martin.v.loewis | 2007-11-20 03:46:02 +0100 (Tue, 20 Nov 2007) | 2 lines

  Patch #1468: Package Lib/test/*.pem.
........
  r59068 | christian.heimes | 2007-11-20 04:21:02 +0100 (Tue, 20 Nov 2007) | 1 line

  Another fix for test_shutil. Martin pointed out that it breaks some build bots
........
  r59073 | nick.coghlan | 2007-11-20 15:55:57 +0100 (Tue, 20 Nov 2007) | 1 line

  Backport some main.c cleanup from the py3k branch
........
  r59076 | amaury.forgeotdarc | 2007-11-21 00:31:27 +0100 (Wed, 21 Nov 2007) | 6 lines

  The incremental decoder for utf-7 must preserve its state between calls.
  Solves issue1460.

  Might not be a backport candidate: a new API function was added,
  and some code may rely on details in utf-7.py.
........
2007-11-20 23:38:09 +00:00
Guido van Rossum
87c0f1d1c9 Merged revisions 59041-59055 via svnmerge from
svn+ssh://pythondev@svn.python.org/python/trunk

........
  r59044 | neal.norwitz | 2007-11-18 17:46:20 -0800 (Sun, 18 Nov 2007) | 1 line

  Use a slightly more recent version than 1.5.2b2.
........
  r59047 | walter.doerwald | 2007-11-19 04:14:05 -0800 (Mon, 19 Nov 2007) | 2 lines

  Fix typo in comment.
........
  r59049 | walter.doerwald | 2007-11-19 04:41:10 -0800 (Mon, 19 Nov 2007) | 4 lines

  Fix for #1444: utf_8_sig.StreamReader was (indirectly through decode())
  calling codecs.utf_8_decode() with final==True, which falled with incomplete
  byte sequences. Fix and test by James G. Sack.
........
  r59051 | nick.coghlan | 2007-11-19 05:56:27 -0800 (Mon, 19 Nov 2007) | 1 line

  Enable some test_cmd_line_script debugging output to investigate failure on Mac OSX buildbot
........
  r59053 | facundo.batista | 2007-11-19 08:30:24 -0800 (Mon, 19 Nov 2007) | 3 lines


  Fixed detail in add_type() explanation (issue 1463).
........
  r59054 | guido.van.rossum | 2007-11-19 09:35:24 -0800 (Mon, 19 Nov 2007) | 2 lines

  Make this work stand-alone, too.
........
  r59055 | guido.van.rossum | 2007-11-19 09:50:22 -0800 (Mon, 19 Nov 2007) | 3 lines

  Fix the OSX failures in this test -- they were due to /tmp being a symlink
  to /private/tmp.  Adding a call to os.path.realpath() to temp_dir() fixed it.
........
2007-11-19 18:03:44 +00:00
Guido van Rossum
98297ee781 Merging the py3k-pep3137 branch back into the py3k branch.
No detailed change log; just check out the change log for the py3k-pep3137
branch.  The most obvious changes:

  - str8 renamed to bytes (PyString at the C level);
  - bytes renamed to buffer (PyBytes at the C level);
  - PyString and PyUnicode are no longer compatible.

I.e. we now have an immutable bytes type and a mutable bytes type.

The behavior of PyString was modified quite a bit, to make it more
bytes-like.  Some changes are still on the to-do list.
2007-11-06 21:34:58 +00:00
Guido van Rossum
3172c5d263 Patch# 1258 by Christian Heimes: kill basestring.
I like this because it makes the code shorter! :-)
2007-10-16 18:12:55 +00:00
Guido van Rossum
04c70ad971 Fix the one failing test (can't decode twice). 2007-08-29 14:04:40 +00:00
Guido van Rossum
09549f4407 Changes in anticipation of stricter str vs. bytes enforcement. 2007-08-27 20:40:10 +00:00
Walter Dörwald
41980caf64 Apply SF patch #1775604: This adds three new codecs (utf-32, utf-32-le and
ut-32-be). On narrow builds the codecs combine surrogate pairs in the unicode
object into one codepoint on encoding and create surrogate pairs for
codepoints outside the BMP on decoding. Lone surrogates are passed through
unchanged in all cases.

Backport to the trunk will follow.
2007-08-16 21:55:45 +00:00