Trivial C-program which includes Python.h does not compile cleanly

Binary package hint: python-dev

Hello!

The following C-program, test.c:

#include <stdlib.h>
#include <Python.h>

int main()
{
  return 0;
}

Does not compile cleanly on Ubuntu 9.10.
Compiling the above program using

gcc -I/usr/include/python2.6 test.c

Is expected to work with no warnings or error messages.
However, instead it yields the following error-message:

/usr/include/python2.6/pyconfig.h:1028:1: warning: "_POSIX_C_SOURCE" redefined
In file included from /usr/include/stdlib.h:25,
                 from t.c:1:
/usr/include/features.h:210:1: warning: this is the location of the previous definition
This is of course 100% repeatable.

Some information about my system:

lsb_release -rd
Description: Ubuntu 9.10
Release: 9.10

libc6: 2.10.1-0ubuntu15
python-dev: 2.6.4-0ubuntu1
gcc: 4.4.4.1-1ubuntu2

From what I understand, the immediate cause of the error is that the Python header files define a few macros to enable certain system features. However, the standard headers in glibc also enable these macros, leading to a macro redefinition warning.

Arguably, the correct fix is that the Python.h shouldn't be unconditionally defining these macros, and that upstream should fix this. However, it appears that this isn't going to happen, see:

http://docs.python.org/c-api/intro.html#includes
http://bugs.python.org/issue793764
(there are many, many more reports)

The following standard is quoted as rationale: http://www.opengroup.org/onlinepubs/009695399/basedefs/xbd_chap02.html .
It says that a *strictly conforming* POSIX application must define the _POSIX_C_SOURCE macro before any system header is included. The Python-headers have taken it upon themselves to do this for the user. If the user does it, Python.h won't work. If the user includes a system header before Python.h, the program is no longer "strictly conforming", the system headers then decide to define _POSIX_C_SOURCE, and Python.h won't work.

In summary:
 * The Python-devs say their header file must be included first, and that this will work on a system supporting strictly conforming POSIX applications.
 * The glibc defines _POSIX_C_SOURCE, if it isn't already defined. (The program is no longer a strictly conforming POSIX application anyway.) Python.h breaks.
 * The Python developers tell us that we must include Python.h first, before *any other header*.
 * Including Python.h first works, but is not an ideal solution. Computer programs often use many different libraries. Only one header file can be included first in any c-file. One of the strengths of Ubuntu is that many different libraries can be used in the same program. As long as only one library requires that its include files be included first, things work. But what if some other package also likes to be included first? The situation isn't sustainable. Python isn't special enough to be able to stipulate that its include files should always be included first.
 * Both the Python-devs and the glibc devs are right. However, we can do better.

My very humble suggestion is that the following patch be applied by Ubuntu for the python-dev package:

--- pyconfig.h.old 2010-01-02 13:40:46.157002791 +0100
+++ pyconfig.h 2010-01-02 14:11:42.155750057 +0100
@@ -1025,7 +1025,12 @@
 /* #undef _OSF_SOURCE */

 /* Define to activate features from IEEE Stds 1003.1-2001 */
+/* NOTE! This has been removed by the Ubuntu maintainer.
+The Ubuntu OS does not require this macro in order to enable
+the latest POSIX features. Redefining it here gives gcc warnings
+about redefined symbols. See bug
 #define _POSIX_C_SOURCE 200112L
+*/

 /* Define if you have POSIX threads, and your system does not define that. */
 /* #undef _POSIX_THREADS */
@@ -1034,7 +1039,12 @@
 /* #undef _REENTRANT */

 /* Define to the level of X/Open that your system supports */
+/* NOTE! This has been removed by the Ubuntu maintainer.
+The Ubuntu OS does not require this macro in order to enable
+the system features. Redefining it here gives gcc warnings
+about redefined symbols.
 #define _XOPEN_SOURCE 600
+*/

 /* Define to activate Unix95-and-earlier features */
 #define _XOPEN_SOURCE_EXTENDED 1

The patch above stops pyconfig.h from defining _POSIX_C_SOURCE. (And also _XOPEN_SOURCE, which is a macro with exactly the same problem).

This patch will never be accepted into upstream Python, since it stops Python from being guaranteed to work on an OS which only supports strictly conforming POSIX Applications (I think). However, we know that in the context of an Ubuntu system, defining this macro isn't necessary! We have more information than the Python-devs, and we can use it.

Now, I understand that this suggestion will be met with skepticism. Altering upstream design decisions shouldn't be done lightly. But perhaps this particular modification is exactly the kind of thing which a Linux distribution should be doing: Making two otherwise incompatible packages (libc6 and python-dev) more compatible?

The effect of the proposed change, as far as I can tell, is:

Advantages:
* Programs can include Python.h and system headers in any order. The two macro definitions removed from pyconfig.h will be defined by the system headers anyway, and system headers are included by Python.h.
* The value of _POSIX_C_SOURCE will be the same for programs regardless of whether they include Python.h. Before this change, you got a lower value of _POSIX_C_SOURCE if you included Python.h. Higher values enable more features, which means that previously you could theoretically lose some POSIX-features if you included Python.h.

Disadvantages:
 * Any existing programs dependent upon the lower value of _POSIX_C_SOURCE defined by Python.h, could conceivably break. These programs can probably almost always be easily fixed by changing them to allow values of _POSIX_C_SOURCE above a certain number, rather than exactly this number.
* A program including pyconfig.h directly, without including Python.h, and without including any system header, and then being dependent on _POSIX_C_SOURCE being set, would no longer work. This seems like a contrived case. Such programs can nonetheless be fixed easily by including for instance <stdlib.h>.

In closing I just want to give a great "Thank You!" to all of the Ubuntu team. You guys are doing great work (regardless of how you respond to this report :-) ).

ProblemType: Bug
Architecture: amd64
Date: Sat Jan 2 13:11:08 2010
DistroRelease: Ubuntu 9.10
NonfreeKernelModules: nvidia
Package: python-dev 2.6.4-0ubuntu1
PackageArchitecture: all
ProcEnviron:
 LANG=en_US.UTF-8
 SHELL=/bin/bash
ProcVersionSignature: Ubuntu 2.6.31-16.53-generic
SourcePackage: python-defaults
Uname: Linux 2.6.31-16-generic x86_64
XsessionErrors:
 (gnome-settings-daemon:2251): GLib-CRITICAL **: g_propagate_error: assertion `src != NULL' failed
 (gnome-settings-daemon:2251): GLib-CRITICAL **: g_propagate_error: assertion `src != NULL' failed
 (polkit-gnome-authentication-agent-1:2288): GLib-CRITICAL **: g_once_init_leave: assertion `initialization_value != 0' failed
 (nautilus:2274): Eel-CRITICAL **: eel_preferences_get_boolean: assertion `preferences_is_initialized ()' failed