pre-PEP : Synthesis of previous threads, and irc talks + proposals

[Tarek Ziadé]

Hello

I have followed most of the threads from the past days, and we talked
a lot on IRC with people from Fedora, Debian, Enthought, TG2 on
possible enhancements
While the other threads are continuing in deeper details, I would like
to start a fresh thread were people don't have to re-read everything
to be able to give their opinions
on very precise points,

This thread is focusing on shouting out the current problems and the
solutions that can be adopted.

I'd like to have "+1" and "-1" on each proposal, with at most one sentence.
or fix a mistake if there is. That could help us speed up the work.

let's try to keep this thread concise, if you want to discuss deeply
on a problem, start another thread, and
i'll follow it to fix my summary.


The problems
===========

1/ the dependencies of a package are not expressed in the Require
metadata of the package most of the time.
   adding a dependency to a module is not really done, developer add
dependencies to packages.

   Furthermore, developer tend to use setuptools "install_requires"
and "tests_require" arguments to express dependencies.

   So basically, you have to run an egg_info to get them, because the
info files are generated by commands.

2/ the existence of PyPI had a side-effect: people tend to push the
entire doc of the package in one single field (long_description)
   to display them at PyPI. The documentation of the package is not
cleary pointed to others.

3/ the metadata infos cannot be expressed in a static file by the
developer, because sometimes they are calculated by code.
    while this very permissive, that is how it works but they are
tighted to argument passing to setup().

4/ PyPI lacks of direct information about dependencies.

    In the meantime, the DOAP project is working on a way to express
dependencies, but it is a work in progress.

5/ ideally, they should be one and only one version of a given package
in an OS-based installation

6/ packagers would like to have more compatibility information to work
out on security upgrades or version conflicts

7/ developers should be able to have more options when they define
version dependencies in their packages, things like:
      A depends on B>=1.2 and B<=2.0  but with a preference to B 1.4
or "avoid B 1.7"

   they give tips to packagers !

8/ the requires-python field is rarely used by people, so unless you
try the package, you don't know when it is a source
    distribution, if it is going to run on various python versions.

9/ unless the developer has a strong comitment to an OS, he will never
create and use a file that is located in /etc

10/ you can't possibily have a complete knowledge of the dependency
graph and possible conflicts when
     you introduce a versioned dependency in your package.

    packages at given versions are known by some people to work well
together or not in a set of versioned packages,
    Let's call this a "known good set" (KGS)

    - OS packager know and maintain the KGS for their distribution.
    - Web framework packagers does it for their application

   you don't. unless you work in a "KGS" environment. But if you want
your package to be a regular python
   package at PyPI, packagers should be able to change its
dependencies to make it fit their own KGS,
  and to build their knowledge on it.

  The developer dependencies infos is a tip and a help for a packager,
not an enforcement. see [7]

 11/ people should always upload the sdist version at PyPI,  they
don't do it always. otherwise it is a pain for packagers.

Proposals
========

this is also a synthezis of what I hurd, and some elements I have
added to respect the needs that were expressed.

0/ a lot of work can be done to clean distutils, no matter what is
decided (another PEP is built for that) cleanning, removing old-style
code, testing

1/ let's change the Python Metadata , in order to introduce a better
dependency system, by

 - officialy introduce "install requires" and "test requires" metadata in there
 - mark "requires" as deprecated

2/ Let's move part of setuptools code in distutils, to respect those changes.

3/ let's create a simple convention : the metadata should be expressed
in a python module called 'pkginfo.py'
   where each metadata is a variable.

   that can be used by setup.py and therefore by any tool that work
with it, even if it does not run
   a setup.py command.

   This is simpler, this is cleaner. you don't have to run some setup
magic to read them.
   at least some magic introduces by commands

4/ let's change PyPI to make it work with the new metadata and to
enforce a few things

Enforcements:
    - a binary distribution cannot be uploaded if a source distrbution
has not been previously provided for the version
    - the requires-python need to be present. : come on, you know what
python versions your package work with !

New features:
   - we should be able to download the metadata of a package without
downloading the package
   - PyPI should display the install and test dependencies in the UI
   - The XML-RPC should provide this new metadata as well.
    - a commenting system should allow developers and packagers to
give more infos on a package at PyPI
     to make the work easier

Open question
============

(please if you want to react on those, open another thread, with a
clean cut, otherwise it is hard to follow directly)

- what about the documentation ? can't we express it better in the
Metadata ? I think we can structurize it a bit

- what about the configuration ? can't we find a way to interact with
a config ini-like file for instance
   and don't care if it is located at /etc/package.cfg or at /Volumes/..etc ?


Regards
Tarek

--
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Fred Drake]

On Wed, Oct 1, 2008 at 8:10 AM, Tarek Ziadé <ziade.tarek@...> wrote:
> 8/ the requires-python field is rarely used by people, so unless you
> try the package, you don't know when it is a source
>    distribution, if it is going to run on various python versions.

What requires-python field?

I don't see this documented for distutils or setuptools.


  -Fred

--
Fred L. Drake, Jr.    <fdrake at gmail.com>
"Chaos is the score upon which reality is written." --Henry Miller
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Jeroen Ruigrok van der Werven-3]

-On [20081001 14:10], Tarek Ziadé (ziade.tarek@...) wrote:
>I have followed most of the threads from the past days, and we talked
>a lot on IRC with people from Fedora, Debian, Enthought, TG2 on
>possible enhancements

Just a note: do not forget the BSD Unix systems when it comes to packaging
and whatnot, it's quite a bit different from the Linux systems. Also,
there's pkgsrc which spans multiple different platforms.

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
イェルーン ラウフロック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
Vae victis!
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Tarek Ziadé]

On Wed, Oct 1, 2008 at 3:08 PM, Fred Drake <fdrake@...> wrote:
> On Wed, Oct 1, 2008 at 8:10 AM, Tarek Ziadé <ziade.tarek@...> wrote:
>> 8/ the requires-python field is rarely used by people, so unless you
>> try the package, you don't know when it is a source
>>    distribution, if it is going to run on various python versions.
>
> What requires-python field?
>
> I don't see this documented for distutils or setuptools.

The one described here http://www.python.org/dev/peps/pep-0345/ in Metadata 1.2

'Requires-Python'

So it can't be used by people at this time, that was a mistake.

the new version of [8] could be:

8/ you don't know when it is a source distribution, if it is going to
run on various python versions.
    PEP 345 defines it, but it is not yet implemented in distutils,
either in setuptools.

and in the proposal parts:

0/ ... + implement the metadata 1.2 from PEP 345, (besides the
"requires"  metadata)

>
>
>  -Fred
>
> --
> Fred L. Drake, Jr.    <fdrake at gmail.com>
> "Chaos is the score upon which reality is written." --Henry Miller
>



--
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Tarek Ziadé]

On Wed, Oct 1, 2008 at 3:16 PM, Jeroen Ruigrok van der Werven
<asmodai@...> wrote:
> -On [20081001 14:10], Tarek Ziadé (ziade.tarek@...) wrote:
>>I have followed most of the threads from the past days, and we talked
>>a lot on IRC with people from Fedora, Debian, Enthought, TG2 on
>>possible enhancements
>
> Just a note: do not forget the BSD Unix systems when it comes to packaging
> and whatnot, it's quite a bit different from the Linux systems. Also,
> there's pkgsrc which spans multiple different platforms.

Yes indeed, a BSD packager could be helpfull in the discussion. Are
you able to help us in this ?


Tarek
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Jeroen Ruigrok van der Werven-3]

-On [20081001 15:24], Tarek Ziadé (ziade.tarek@...) wrote:
>Yes indeed, a BSD packager could be helpfull in the discussion. Are
>you able to help us in this ?

I used to be a FreeBSD and DragonFly BSD committer and I still use Python
mostly on BSD systems. I am sure Trent Nelson could also help. I know Joerg
Sonnenberger from pkgsrc/NetBSD quite well too.

So all in all I should be able to.

My time seems a bit more limited than yours though. :)

--
Jeroen Ruigrok van der Werven <asmodai(-at-)in-nomine.org> / asmodai
イェルーン ラウフロック ヴァン デル ウェルヴェン
http://www.in-nomine.org/ | http://www.rangaku.org/ | GPG: 2EAC625B
Dreams are like Angels, they keep bad at bay, bad at bay, Love is the
Light, scaring Darkness away...
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Phillip J. Eby]

At 02:10 PM 10/1/2008 +0200, Tarek Ziadé wrote:

>Proposals
>========
>
>this is also a synthezis of what I hurd, and some elements I have
>added to respect the needs that were expressed.
>
>0/ a lot of work can be done to clean distutils, no matter what is
>decided (another PEP is built for that) cleanning, removing old-style
>code, testing
>
>1/ let's change the Python Metadata , in order to introduce a better
>dependency system, by
>
>  - officialy introduce "install requires" and "test requires"
> metadata in there
>  - mark "requires" as deprecated
>
>2/ Let's move part of setuptools code in distutils, to respect those changes.
>
>3/ let's create a simple convention : the metadata should be expressed
>in a python module called 'pkginfo.py'
>    where each metadata is a variable.
>
>    that can be used by setup.py and therefore by any tool that work
>with it, even if it does not run
>    a setup.py command.
>
>    This is simpler, this is cleaner. you don't have to run some setup
>magic to read them.
>    at least some magic introduces by commands

I'm -1 on all of the above.  I think we need a standard for tools
interop (ala WSGI), not implementation tweaks for the existing
tools.  I also think that a concrete metadata format proposal is
premature at this time; we've barely begun to gather -- let alone
specify -- our requirements for that metadata.  (Essentially, only
version dependencies have been discussed, AFAICT.)

There have been many people agreeing that the distutils are
thoroughly broken and a new approach is needed; these  proposals
sound like minor tweaks to the existing infrastructure, rather than a
way to get rid of it.  So to me, the above doesn't seem like a
synthesis of the threads that I've been reading.


>4/ let's change PyPI to make it work with the new metadata and to
>enforce a few things
>
>Enforcements:
>     - a binary distribution cannot be uploaded if a source distrbution
>has not been previously provided for the version

Note that this doesn't allow closed-source packages to be uploaded;
thus it would need to be a warning, rather than a requirement.


>New features:
>    - we should be able to download the metadata of a package without
>downloading the package
>    - PyPI should display the install and test dependencies in the UI

It could only do this for specific binaries, since dependencies can be dynamic.

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Tarek Ziadé]

On Wed, Oct 1, 2008 at 7:29 PM, Phillip J. Eby <pje@...> wrote:
> I'm -1 on all of the above.  I think we need a standard for tools interop
> (ala WSGI), not implementation tweaks for the existing tools.  I also think
> that a concrete metadata format proposal is premature at this time; we've
> barely begun to gather -- let alone specify -- our requirements for that
> metadata.  (Essentially, only version dependencies have been discussed,
> AFAICT.)

What are the other important points we need to discuss at this point
in your opinion ?

>
> There have been many people agreeing that the distutils are thoroughly
> broken and a new approach is needed; these  proposals sound like minor
> tweaks to the existing infrastructure, rather than a way to get rid of it.
>  So to me, the above doesn't seem like a synthesis of the threads that I've
> been reading.
>
>
>> 4/ let's change PyPI to make it work with the new metadata and to
>> enforce a few things
>>
>> Enforcements:
>>    - a binary distribution cannot be uploaded if a source distrbution
>> has not been previously provided for the version
>
> Note that this doesn't allow closed-source packages to be uploaded; thus it
> would need to be a warning, rather than a requirement.
>

Right. do you agree it is something useful to do ?

>
>> New features:
>>   - we should be able to download the metadata of a package without
>> downloading the package
>>   - PyPI should display the install and test dependencies in the UI
>
> It could only do this for specific binaries, since dependencies can be
> dynamic.
>
>

What dynamic means here ? the python module to static file process or more ?
can you provide an example ?

Regards
Tarek

--
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Phillip J. Eby]

At 07:55 PM 10/1/2008 +0200, Tarek Ziadé wrote:

>On Wed, Oct 1, 2008 at 7:29 PM, Phillip J. Eby <pje@...> wrote:
> > I'm -1 on all of the above.  I think we need a standard for tools interop
> > (ala WSGI), not implementation tweaks for the existing tools.  I also think
> > that a concrete metadata format proposal is premature at this time; we've
> > barely begun to gather -- let alone specify -- our requirements for that
> > metadata.  (Essentially, only version dependencies have been discussed,
> > AFAICT.)
>
>What are the other important points we need to discuss at this point
>in your opinion ?

What information needs to be conveyed by a "build" tool to an
"install" tool, and vice versa.

For example, an install tool needs to know what files are
documentation, which are sample data, and what is part of the library
proper, as well as what things are scripts (and in what language
those scripts are written, e.g. Python, shell, batch, etc.).  Some
install tools need to know about icons, menus, registry info, cron
jobs, etc.   (These are perhaps more properly the domain of
applications than libraries, but I'm going to assume that these
things are in scope.)

The way that this information is communicated needs to be extensible,
so that optional metadata for debs and msi's and rpm's and
whathaveyou can be incorporated, without needing to modify the
standard -- especially if the APIs for reading and writing this data
are in the stdlib.

There needs to be a way for install tools to ask a source package to
"configure" itself, possibly specifying options (and a way for it to
find out what those options are, to be able to present them to the
user).  Then there needs to be a way for the install tool to ask the
package to build itself with the configured options, and a standard
for how the build tool(s) communicate errors or other issues back to
the install tools.

There needs to be a way, of course, for the package to specify what
build tools it needs in order to be built, and for those to plug in
to the (again stdlib-contained) build API.

There needs to be a better API for querying C configuration and
compiler details, that's separate from the distutils "ccompiler" stuff.

Last, but not least, there needs to be a definition of core build and
install tools to be both an example/reference implementation of the
standard, and to provide the basics needed by the core.

I think I've mentioned all of these previously in the thread.  I also
think that as a matter of technicalities, these things are not
difficult to achieve...  but if it they are only achieved
*technically*, then the result is a failure, not a success.

In order for the *real* goal to be achieved (i.e., a flourishing
build/install system for Python), widespread participation and buy-in
is required.  If the OS people or the big package people (e.g. Zope
Corp., Enthought) or the distutils aficionados are left out, then the
result won't get used.

I think the best way to ensure that nobody is left out, is to get
them to participate in the design of a standard that ensures that
*they* will be able to control their destiny, by creating their own
build plugins and/or install tools...  or at least having a robust
choice of alternatives.

We need a consensus "de jure" standard (ala WSGI), rather than just
an ongoing "de facto" standard (ala distutils/setuptools), or we're
not making any substantial progress, just handing the reins over to
somebody else.


> >> Enforcements:
> >>    - a binary distribution cannot be uploaded if a source distrbution
> >> has not been previously provided for the version
> >
> > Note that this doesn't allow closed-source packages to be uploaded; thus it
> > would need to be a warning, rather than a requirement.
> >
>
>Right. do you agree it is something useful to do ?

Sure.

> >> New features:
> >>   - we should be able to download the metadata of a package without
> >> downloading the package
> >>   - PyPI should display the install and test dependencies in the UI
> >
> > It could only do this for specific binaries, since dependencies can be
> > dynamic.
> >
> >
>
>What dynamic means here ? the python module to static file process or more ?
>can you provide an example ?

Dependencies can be platform-specific as well as python-version
specific.  If you want ctypes, you would depend on it in python 2.4
but not in 2.5.  Similarly, if on some platform a certain library is
required to implement a feature that is natively accessible on other
platforms.  In these cases, you would have logic in setup.py to
detect this and choose the appropriate dependencies.

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Gael Varoquaux-2]

On Wed, Oct 01, 2008 at 02:28:07PM -0400, Phillip J. Eby wrote:

> In order for the *real* goal to be achieved (i.e., a flourishing
> build/install system for Python), widespread participation and buy-in is
> required.  If the OS people or the big package people (e.g. Zope Corp.,
> Enthought) or the distutils aficionados are left out, then the result won't
> get used.
>
> [...]
>
> We need a consensus "de jure" standard (ala WSGI), rather than just an
> ongoing "de facto" standard (ala distutils/setuptools), or we're not making
> any substantial progress, just handing the reins over to somebody else.

Nice words. I like very much the sound of this discussion.

Ga�l
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Tarek Ziadé]

On Wed, Oct 1, 2008 at 8:28 PM, Phillip J. Eby <pje@...> wrote:

> At 07:55 PM 10/1/2008 +0200, Tarek Ziadé wrote:
>>
>> On Wed, Oct 1, 2008 at 7:29 PM, Phillip J. Eby <pje@...>
>> wrote:
>> > I'm -1 on all of the above.  I think we need a standard for tools
>> > interop
>> > (ala WSGI), not implementation tweaks for the existing tools.  I also
>> > think
>> > that a concrete metadata format proposal is premature at this time;
>> > we've
>> > barely begun to gather -- let alone specify -- our requirements for that
>> > metadata.  (Essentially, only version dependencies have been discussed,
>> > AFAICT.)
>>
>> What are the other important points we need to discuss at this point
>> in your opinion ?
>
> What information needs to be conveyed by a "build" tool to an "install"
> tool, and vice versa.
>
> For example, an install tool needs to know what files are documentation,
> which are sample data, and what is part of the library proper, as well as
> what things are scripts (and in what language those scripts are written,
> e.g. Python, shell, batch, etc.).  Some install tools need to know about
> icons, menus, registry info, cron jobs, etc.   (These are perhaps more
> properly the domain of applications than libraries, but I'm going to assume
> that these things are in scope.)
>
> The way that this information is communicated needs to be extensible, so
> that optional metadata for debs and msi's and rpm's and whathaveyou can be
> incorporated, without needing to modify the standard -- especially if the
> APIs for reading and writing this data are in the stdlib.

So basically with this system, this would mean that an ini-file in my package
would be marked as a configuration file for the installer API, and that third
party tools would decide what to do with this file.

This would mean that my program would have to access to that file
through an API as well to get back to the ini-file in the system.

how would it work ?

> I think I've mentioned all of these previously in the thread.  I also think
> that as a matter of technicalities, these things are not difficult to
> achieve...  but if it they are only achieved *technically*, then the result
> is a failure, not a success.
>
> In order for the *real* goal to be achieved (i.e., a flourishing
> build/install system for Python), widespread participation and buy-in is
> required.  If the OS people or the big package people (e.g. Zope Corp.,
> Enthought) or the distutils aficionados are left out, then the result won't
> get used.

Well yes, that is basically what we are trying to build since a few
days, but threads are
not linear so people cant' keep up and jump in like that.

So maybe you mentioned that idea before in the thread,
but if we want people to buy the idea, it should be put in the wiki
imho, even prematurely,
and built it there, starting from a small set of points.

I mean, you said earlier that it is premature to write a concrete document
but it's hard to follow in threads the ideas proposed. That was the
idea of my early proposal : start a synthetic list of problems
and for each problem possible solutions. Then discuss them in the ML and
make the right one raise.

That is just my 2 cents on how the discussions go


>
> I think the best way to ensure that nobody is left out, is to get them to
> participate in the design of a standard that ensures that *they* will be
> able to control their destiny, by creating their own build plugins and/or
> install tools...  or at least having a robust choice of alternatives.
>
> We need a consensus "de jure" standard (ala WSGI), rather than just an
> ongoing "de facto" standard (ala distutils/setuptools), or we're not making
> any substantial progress, just handing the reins over to somebody else.

+10

>
>
>> >> Enforcements:
>> >>    - a binary distribution cannot be uploaded if a source distrbution
>> >> has not been previously provided for the version
>> >
>> > Note that this doesn't allow closed-source packages to be uploaded; thus
>> > it
>> > would need to be a warning, rather than a requirement.
>> >
>>
>> Right. do you agree it is something useful to do ?
>
> Sure.

Ok so maybe this is *one* problem we can already solve at PyPI with a patch.

>> >>   - PyPI should display the install and test dependencies in the UI
>> >
>> > It could only do this for specific binaries, since dependencies can be
>> > dynamic.
>> >
>> >
>>
>> What dynamic means here ? the python module to static file process or more
>> ?
>> can you provide an example ?
>
> Dependencies can be platform-specific as well as python-version specific.
>  If you want ctypes, you would depend on it in python 2.4 but not in 2.5.
>  Similarly, if on some platform a certain library is required to implement a
> feature that is natively accessible on other platforms.  In these cases, you
> would have logic in setup.py to detect this and choose the appropriate
> dependencies.

ok, right it is not possible with what we have now.

If each dependency was marked as platform-specific or python-version specific
in the metadata when it is necessary,  we would know them all without calling
extra detection code to build them.

I hate the idea of dynamic metadata in fact. I can't express precisely
why at that point.

>
>



--
Tarek Ziadé | Association AfPy | www.afpy.org
Blog FR | http://programmation-python.org
Blog EN | http://tarekziade.wordpress.com/
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Chris Withers]

Phillip J. Eby wrote:
> I think we need a standard for tools
> interop (ala WSGI), not implementation tweaks for the existing tools.  

Agreed.

>> 4/ let's change PyPI to make it work with the new metadata and to
>> enforce a few things
>>
>> Enforcements:
>>     - a binary distribution cannot be uploaded if a source distrbution
>> has not been previously provided for the version
>
> Note that this doesn't allow closed-source packages to be uploaded; thus
> it would need to be a warning, rather than a requirement.

This is an important point. We can't assume any one repository will have
all needed packages.

> It could only do this for specific binaries, since dependencies can be
> dynamic.

They should not be dynamic :-(

Chris

--
Simplistix - Content Management, Zope & Python Consulting
            - http://www.simplistix.co.uk
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[Phillip J. Eby]

At 04:18 PM 10/3/2008 +0100, Chris Withers wrote:
>Phillip J. Eby wrote:
>>It could only do this for specific binaries, since dependencies can
>>be dynamic.
>
>They should not be dynamic :-(

Too bad. They are, because they have to be in order to support more
than one platform and/or Python version from the same source base.

_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig

[zooko]

Thanks for the synthesis, Tarek.  I have some experience using  
current Python packaging in the field -- the Tahoe project [1] -- and  
so I would like to throw in what I know of what is currently working  
and what is currently needed and what isn't a big deal to me.

This doesn't, of course, mean that other people might value things  
that I don't, but at least the following opinions of mine are won  
from hard experience.

On Oct 1, 2008, at 6:10 AM, Tarek Ziadé wrote:

> 1/ the dependencies of a package are not expressed in the Require
> metadata of the package most of the time.

+2 -- This is the biggest problem.  The dependencies are not  
expressed *anywhere* in the metadata of the package most of the  
time.  We need a de jure and de facto way to express dependencies so  
that developers will actually write them down.

>    Furthermore, developer tend to use setuptools "install_requires"
> and "tests_require" arguments to express dependencies.
>
>    So basically, you have to run an egg_info to get them, because the
> info files are generated by commands.

+0 -- I can see how this could be done better, but it isn't a  
pressing problem us.  The current mechanism to get that dependency  
information at build/develop/install time works okay.

> 2/ the existence of PyPI had a side-effect: people tend to push the
> entire doc of the package in one single field (long_description)
>    to display them at PyPI. The documentation of the package is not
> cleary pointed to others.

+0 -- I would like more structured docs because then I could patch  
stdeb [2] to put docs into /usr/share/docs/$PACKAGE on Debian.  But  
it isn't a pressing problem for us (we currently kludge around that  
issue).

> 3/ the metadata infos cannot be expressed in a static file by the
> developer, because sometimes they are calculated by code.
>     while this very permissive, that is how it works but they are
> tighted to argument passing to setup().

+0 -- I totally agree that a static, separate, declarative file  
containing just data and no code would be a nicer way to do this.  
But the current way is working for us.

> 4/ PyPI lacks of direct information about dependencies.

+? -- I don't know.  It sounds like it would be a big improvement,  
but the current mechanism of discovering dependencies by downloading  
distributions and executing their setup.py's seems to be working.

> 5/ ideally, they should be one and only one version of a given package
> in an OS-based installation

-1 -- This is the strong preference of the folks who package software  
for OSes -- Debian, Fedora, etc. -- but it is not necessarily the  
choice of the users who use their OSes.  It is best for the Python  
packaging standards to be agnostic towards this, or at least to  
support both this desideratum and its opposite.

> 6/ packagers would like to have more compatibility information to work
> out on security upgrades or version conflicts
> 7/ developers should be able to have more options when they define
> version dependencies in their packages, things like:
>       A depends on B>=1.2 and B<=2.0  but with a preference to B 1.4
> or "avoid B 1.7"
>
>    they give tips to packagers !

+0 -- If we try to do better than Debian and Fedora already do then  
this risks being a science project -- i.e. something that will take a  
few years and might or might not pan out.  If we try to just ape them  
and learn from their decade's worth of mistakes then this is probably  
doable.

>   The developer dependencies infos is a tip and a help for a packager,
> not an enforcement. see [7]

+1 -- In around 95% of the cases that I've seen, the developer's  
dependencies info was good enough.  But, people have to be able to do  
something about the other 5%, so they have to be able to override  
developer-provided dependency information with their own.  Obviously  
they can do this by patching or runtime-patching or maintaining their  
own branch, but we should specify a standard, principled way to do it  
instead.

>  11/ people should always upload the sdist version at PyPI,  they
> don't do it always. otherwise it is a pain for packagers.

+1 -- sdist format should be encouraged.


> 1/ let's change the Python Metadata , in order to introduce a better
> dependency system, by
>
>  - officialy introduce "install requires" and "test requires"  
> metadata in there
>  - mark "requires" as deprecated

+1

> 2/ Let's move part of setuptools code in distutils, to respect  
> those changes.

+1

> 3/ let's create a simple convention : the metadata should be expressed
> in a python module called 'pkginfo.py'
>    where each metadata is a variable.
>
>    that can be used by setup.py and therefore by any tool that work
> with it, even if it does not run
>    a setup.py command.
>
>    This is simpler, this is cleaner. you don't have to run some setup
> magic to read them.
>    at least some magic introduces by commands

Uh...  I thought the idea was to *not* have arbitrary Python code  
executed in this part?  How about a flat file that people can  
reliably parse with, say, "grep", to learn about metadata.

>     - a binary distribution cannot be uploaded if a source distrbution
> has not been previously provided for the version
>     - the requires-python need to be present. : come on, you know what
> python versions your package work with !

+1

>    - we should be able to download the metadata of a package without
> downloading the package
>    - PyPI should display the install and test dependencies in the UI
>    - The XML-RPC should provide this new metadata as well.
>     - a commenting system should allow developers and packagers to
> give more infos on a package at PyPI
>      to make the work easier

+1

Regards,

Zooko

[1] http://allmydata.org/trac/tahoe
[2] http://stdeb.python-hosting.com/

---
http://allmydata.org -- Tahoe, the Least-Authority Filesystem
http://allmydata.com -- back up all your files for $5/month
_______________________________________________
Distutils-SIG maillist  -  Distutils-SIG@...
http://mail.python.org/mailman/listinfo/distutils-sig