[issue4051] use of TCHAR under win32

[David W. Lambert]

New submission from Ulrich Eckhardt <eckhardt@...>:

In several places, Python is using TCHAR-based win32 APIs which has
several consequences:
1. The actually used function depends on the _UNICODE macro, while the
using code often assumes that this macro is not defined. That means
that not only the code changes when the macro is defined but that it
simply breaks.
2. Using the ANSI-variants of functions causes overhead when a Unicode
string has to be transcoded and possibly data loss, because the ANSI
encoding can't represent the full Unicode range. Note that I'm
assuming an NT based system (i.e. NT, win2k etc), because the systems
where the ANSI encoding is native (win9x, ME(?) etc) have been dropped
from the supported platforms.
3. The main reason for me wanting a build independent of TCHAR is a
port to MS Windows CE, which doesn't support the ANSI encoding at all.
This means that a TCHAR-clean build for desktop win32 is a necessary
part of the port to CE, but even without the port it would be useful
to skip the conversion overhead and at the same time gain
functionality. Concerning the CE port, see this thread as a reference:
http://mail.python.org/pipermail/python-dev/2008-September/082529.html

Note: I'm adding this tracker item mainly in order to attach patches
for review, so they don't get lost like it's possible on the
mailinglist.

----------
components: Windows
messages: 74363
nosy: eckhardt
severity: normal
status: open
title: use of TCHAR under win32
versions: Python 2.7

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4051>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[David W. Lambert]

Ulrich Eckhardt <eckhardt@...> added the comment:

Add a patch that fixes a warning.

----------
keywords: +patch
Added file: http://bugs.python.org/file11708/Python-unicode-redefine-warning.0.patch

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4051>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[David W. Lambert]

Martin v. Löwis <martin@...> added the comment:

Please create a separate tracker item per individual change (or per
group by enourmously similar changes). Each change will be reviewed
separately, and will get accepted or rejected separately.

This specific patch (Python-unicode-redefine-warning) is fine, but I'd
rather wait until after the release of Python 3.0, in order to be able
to merge the changes better into the 3k branch.

----------
nosy: +loewis

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4051>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[David W. Lambert]

Roumen Petrov <bugtrack@...> added the comment:

What about to substitute UNICODE macro as example with RAWUNICODE in the
source ?

----------
nosy: +rpetrov

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4051>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[David W. Lambert]

Martin v. Löwis <martin@...> added the comment:

> What about to substitute UNICODE macro as example with RAWUNICODE in the
> source ?

That would work as well. As it stands, the macro names are equivalent to
the ones in pickle.py, which is a useful property to have. So I'm in
favor of the proposed change.

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4051>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[David W. Lambert]

Roumen Petrov <bugtrack@...> added the comment:

For the protocol I attach part for PythonCE-2.5-20061219.patch. Both
patches address UNICODE name clash in Modules/cPickle.c.

For windows programs UNICODE is more commonly used define and may be
python C-code only has to avoid issues like this one.

I couldn't found where python C-code use so called "TCHAR meaning" and
the first sentence from comment in Ulrich patch looks irrelevant to the
issue.

Also Ulrich patch is far from complete "TCHAR" implementation in python.
May be issue title has to be changed to address more precise problem.

Added file: http://bugs.python.org/file11768/PythonCE-2.5-20061219-cPickle.c-patch

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4051>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[David W. Lambert]

Martin v. Löwis <martin@...> added the comment:

> I couldn't found where python C-code use so called "TCHAR meaning" and
> the first sentence from comment in Ulrich patch looks irrelevant to the
> issue.

I was confused by Ulrich's terminology at first, also (claiming that
Python doesn't use TCHAR, so how could it depend on it); I now think
Ulrich is nearly correct. E.g. in 2.6, we call, in posixmodule.c,
CreateProcess. Now, CreateProcess is defined in the SDK as

#ifdef UNICODE
#define CreateProcess  CreateProcessW
#else
#define CreateProcess  CreateProcessA
#endif // !UNICODE

However, it is *documented* as

BOOL WINAPI CreateProcess(
  __in_opt     LPCTSTR lpApplicationName,
  __inout_opt  LPTSTR lpCommandLine,
  __in_opt     LPSECURITY_ATTRIBUTES lpProcessAttributes,
  __in_opt     LPSECURITY_ATTRIBUTES lpThreadAttributes,
  __in         BOOL bInheritHandles,
  __in         DWORD dwCreationFlags,
  __in_opt     LPVOID lpEnvironment,
  __in_opt     LPCTSTR lpCurrentDirectory,
  __in         LPSTARTUPINFO lpStartupInfo,
  __out        LPPROCESS_INFORMATION lpProcessInformation
);

So according to the documentation, this is a function that takes TCHAR
parameters, hence it is a "TCHAR-based (windows) API". The fact that the
Windows SDK actually defines it differently is a technical hack; for all
practical purposes, it behaves as if it really was a TCHAR API.

> Also Ulrich patch is far from complete "TCHAR" implementation in python.

See msg74386.

> May be issue title has to be changed to address more precise problem.

Certainly.

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4051>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com

[David W. Lambert]

Changes by Martin v. Löwis <martin@...>:


----------
title: use of TCHAR under win32 -> UNICODE macro in cPickle conflicts with Windows define

_______________________________________
Python tracker <report@...>
<http://bugs.python.org/issue4051>
_______________________________________
_______________________________________________
Python-bugs-list mailing list
Unsubscribe: http://mail.python.org/mailman/options/python-bugs-list/lists%40nabble.com