Debian-science: synthesis about VCS & repository structure

[Sylvestre Ledru-3]

Hello,

After many emails and interesting discussions, it is time to make a
synthesis.

VCS
---
Keeping two repositories will be confusing, harder to maintain and a bit
non-sense.
Since nobody complained about git choice, I proposed that git becomes
the only VCS that debian-science uses.

Repository structure
--------------------
Since there are many concerns about the "category structure" and the
structure proposed by Manuel [1] seems a good one:

-- debian science
   +-- homepage => the website of Debian-science
   +-- packages => packages
        +-- bar
        +-- foo
   +-- policy => documents describing the debian-science
   +-- taskss => Package "work in progress"
   +-- ...

At the moment, there is no need to split packages by letter, it will
only create a feeling of emptiness. But when we will reach ~ 100
packages (and I am sure we will), we could reconsider this.

If there is no strong objections, I will migrate the current svn to git.

Sylvestre


[1] http://lists.debian.org/debian-science/2008/05/msg00142.html



--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[smr99]

On Mon, May 19, 2008 at 11:13:18AM +0200, Sylvestre Ledru wrote:

> VCS
> ---
> Keeping two repositories will be confusing, harder to maintain and a bit
> non-sense.
> Since nobody complained about git choice, I proposed that git becomes
> the only VCS that debian-science uses.

Wait a minute.  What's wrong with using svn for those who want to?
I have some packages that I'd like to put into debian-science at some
point, but I'm not about to learn the favourite vcs of the week to
do so.

> Repository structure
> --------------------
> Since there are many concerns about the "category structure" and the
> structure proposed by Manuel [1] seems a good one:
>
> -- debian science
>    +-- homepage => the website of Debian-science
>    +-- packages => packages
>         +-- bar
>         +-- foo
>    +-- policy => documents describing the debian-science
>    +-- taskss => Package "work in progress"
>    +-- ...
>
> At the moment, there is no need to split packages by letter, it will
> only create a feeling of emptiness. But when we will reach ~ 100
> packages (and I am sure we will), we could reconsider this.

I'm not sure why you'd want to split it even with 100 packages.  It
would be a bit of a nuisance working out the URL to use.  The
splitting is done in the debian archive, as I understand it, really as
a hack around inefficient directory lookup.  But the archive would
have thousands of entries, not hundreds.

-Steve

[Andreas Tille]

On Mon, 19 May 2008, Steve M. Robbins wrote:

> Wait a minute.  What's wrong with using svn for those who want to?
> I have some packages that I'd like to put into debian-science at some
> point, but I'm not about to learn the favourite vcs of the week to
> do so.

"favourite vcs of the week" -> ROFL ;-)

I definitely share your point about the pet VCSes that are growing all
over Free Software world one performing better as the other.  On the other
hand according to my observation git gains some position inside Debian
to become something like _the_ VCS and thus I think we should try to
settle on a reasonable default.  This could be supported by a well
written policy that contains step by step examples which might be a
good help for newcommers.  If this is done I see using git more as a
chance to learn something new and valuable for my day to day work than
a nuisance to follow the habits of some geeks.  So I havily relay on
a really good documentation what to do.

Kind regards

          Andreas.

--
http://fam-tille.de


--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[David Bremner-4]

>>>>> "Sylvestre" == Sylvestre Ledru <sylvestre.ledru@...> writes:

    Sylvestre> If there is no strong objections, I will migrate the
    Sylvestre> current svn to git.

My experience is that the migration is not quite automatic, at least
if you want the result to work nicely with git-buildpackage (which by
the way I think should probably be part of our "suggested work flow").

Because of this I think it is better to have maintainers migrate their
own packages.  If you want to facilitate this process, a HOWTO would
be very good, and useful not just for this transition.

David


--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Manuel Prinz-2]

Sylvestre, thanks for summing up! Just have my two cents about that:

Am Montag, den 19.05.2008, 11:13 +0200 schrieb Sylvestre Ledru:
> VCS
> ---
> Keeping two repositories will be confusing, harder to maintain and a bit
> non-sense.
> Since nobody complained about git choice, I proposed that git becomes
> the only VCS that debian-science uses.

I have to say that I do not see a reason why to enforce things here. Of
course it's not the optimal solution but I would be perfectly fine to
have both system in parallel so people have time to learn a new VCS. I
think we should recommend using Git, not enforcing it, unless there are
some serious problems (maintainance burden, technical problems,
whatever). (I've not heared from anyone wanting back to SVN once
familiar with Git.)

> Repository structure
> --------------------
> Since there are many concerns about the "category structure" and the
> structure proposed by Manuel [1] seems a good one:
>
> -- debian science
>    +-- homepage => the website of Debian-science
>    +-- packages => packages
>         +-- bar
>         +-- foo
>    +-- policy => documents describing the debian-science
>    +-- taskss => Package "work in progress"

         ^^^^^

         This is what I was talking about in my previous emails: I do
not see it as a directory for "work in progress" packages but as a
directory containing task descriptions, as the Science CDD does! This is
the meta-information I was talking about: The task/category is extracted
from those files, so one can add a package to the "physics" and
"chemistry" files. In that way, a package fits into two categories,
without having it to depend on any directory structure. I think we can
use tools already existing for the Science CDD / Debian-Med. (If they
allow us to use those. ;) )

From those files/descriptions, meta-packages can be build, or they can
even be used for a CDD. (Not familiar with CDDs yet, so just
speculating.) In short: The tasks directory is the place where
task/categorization information goes.

Best regards
Manuel

[Manuel Prinz-2]

Am Montag, den 19.05.2008, 12:20 +0200 schrieb Andreas Tille:
> This [Git being Debian's favorite VCS] could be supported by a well
> written policy that contains step by step examples which might be a
> good help for newcommers.  If this is done I see using git more as a
> chance to learn something new and valuable for my day to day work than
> a nuisance to follow the habits of some geeks.  So I havily relay on
> a really good documentation what to do.

I started working on this document which hopefully will grow to a group
policy by the help of others. It's not much yet but I plan to make it
accessible soon (during the next two days) so everyone can participate;
I just wanted to start it because I think with an existing frame
everyone can be more productive; I do not want to dictate it and hope I
can find some more people willing to write documentation. (It sucks at
times, I know.)

I'm not quite sure if documentation about how to use tools should be in
a policy-like document. But I also think that having in a second
document may not be optimal too. So I'd propose to add those description
to the appendix, so it stays in the policy document and can be accessed
by those interested. Or should we create an "Debian Science Maintainer's
Guide"? (I'm not very creative in inventing names, I hope it's clear
what I mean.)

Best regards
Manuel

[Andreas Tille]

On Mon, 19 May 2008, Manuel Prinz wrote:

> I'm not quite sure if documentation about how to use tools should be in
> a policy-like document.

Right.  Even if some SVN / Quilt usage is in the Debian Med policy I
admit that this is not really a typical content for a policy.  So I
do not care whether the document that describes the usage of GIT is
called "policy" or "Tips for git newbees" or "What you always wanted to
ask about git ..." - as long as there is a simple step by step introduction
I could use.

> But I also think that having in a second document may not be optimal too.

This was catually the reason for including the stuff in the Debian Med
policy - it was just a pragmatical decision which finally goes farther
than a policy usually does.

> So I'd propose to add those description
> to the appendix, so it stays in the policy document and can be accessed
> by those interested. Or should we create an "Debian Science Maintainer's
> Guide"? (I'm not very creative in inventing names, I hope it's clear
> what I mean.)

It's perfectly clear what you mean and both suggestions are perfectly
valid in my eyes.

Kind regards

        Andreas.

--
http://fam-tille.de


--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Andreas Tille]

On Mon, 19 May 2008, Manuel Prinz wrote:

> I have to say that I do not see a reason why to enforce things here. Of
> course it's not the optimal solution but I would be perfectly fine to
> have both system in parallel so people have time to learn a new VCS.

If you have to much time to learn it might be longer than your life time
(even if you become >=100 years old).  I know from own experience that
I would try to safe my time and use the old tool I know as long as it is
possible.  The best chance to start with a new tool is a new project.
Once you insert data with the old toll and try the "I could move this
stuff later anyway" you will not do so because in most cases moving
data from an existing system to another one is much harder than start
from scratch.

> I think we should recommend using Git, not enforcing it, unless there are
> some serious problems (maintainance burden, technical problems,
> whatever). (I've not heared from anyone wanting back to SVN once
> familiar with Git.)

I've heard from noone wanting back to Word once familiar with LaTeX.
If more people would have started with LaTeX the number of copies of
Word would be much smaller ...

>         This is what I was talking about in my previous emails: I do
> not see it as a directory for "work in progress" packages but as a
> directory containing task descriptions, as the Science CDD does! This is
> the meta-information I was talking about: The task/category is extracted
> from those files, so one can add a package to the "physics" and
> "chemistry" files. In that way, a package fits into two categories,
> without having it to depend on any directory structure. I think we can
> use tools already existing for the Science CDD / Debian-Med. (If they
> allow us to use those. ;) )

It is the declared *intention* that these tools are widely used and
I would like to encourage everybody to do so.  Currently the tasks
files are stored in the CDD repository under projects and I think this
is a reasonable place to store them.  Everybody who is interested to
enhance this stuff is very welcome - just ask me to be added to the
alioth team.  I would like to suggest the debian-custom mailing list
which is intended to discuss common CDD issues.

> From those files/descriptions, meta-packages can be build, or they can
> even be used for a CDD. (Not familiar with CDDs yet, so just
> speculating.) In short: The tasks directory is the place where
> task/categorization information goes.

As I said: It exist in another place which turned out to be a not so
bad place.  If there are good reasons to move this stuff to the
debian-science repository this is fine for me as well.  The idea
to store them in a common VCS with debian-med etc was to keep them
together to let the tools work on a common place - but this is not
really necessary.

Kind regards

         Andreas.

--
http://fam-tille.de


--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[smr99]

On Mon, May 19, 2008 at 12:20:48PM +0200, Andreas Tille wrote:

> On Mon, 19 May 2008, Steve M. Robbins wrote:
>
>> Wait a minute.  What's wrong with using svn for those who want to?
>> I have some packages that I'd like to put into debian-science at some
>> point, but I'm not about to learn the favourite vcs of the week to
>> do so.
>
> "favourite vcs of the week" -> ROFL ;-)
>
> I definitely share your point about the pet VCSes that are growing all
> over Free Software world one performing better as the other.  On the other
> hand according to my observation git gains some position inside Debian
> to become something like _the_ VCS and thus I think we should try to
> settle on a reasonable default.  

I have no issue at all with git being the default.  While I don't know
anything about git, a lot of smart people are using & praising it so
I'm sure it does offer myriad advantages over other systems.  At least
for them.

I was merely reacting to Sylvestre's proposal that git be the ONLY vcs
allowed.  My question stands: is there any real disadvantage to
allowing the user's choice of vcs?

The main disadvantage I can think of is that it is a nuisance for
those who want to do project-wide work; for example, David Paleino
once set up a watch file for insighttoolkit.  My response to that
objection is that (at least in my experience with debian-med) the
fraction of project-wide work is trivial and if I choose the minority
vcs, I'll live with not having the attention.

The main ADvantage of allowing several vcs is inclusivity.  You well
know that some refuse to use debian-med because it allows only SVN.  I
hope debian-science won't repeat that mistake.


> This could be supported by a well
> written policy that contains step by step examples which might be a
> good help for newcommers.  If this is done I see using git more as a
> chance to learn something new and valuable for my day to day work than
> a nuisance to follow the habits of some geeks.

Yes, I do appreciate beginner docs; I relied on them for learning svn
and quilt.  I'm not adverse to learning git; but I will do it on my
terms, not in order to join debian-science.

Regards,
-Steve

[Andreas Tille]

On Mon, 19 May 2008, Steve M. Robbins wrote:

> I was merely reacting to Sylvestre's proposal that git be the ONLY vcs
> allowed.  My question stands: is there any real disadvantage to
> allowing the user's choice of vcs?

I think there is.  In the Debian Med team people sometimes do some changes
via script in the whole repository if there are some systematical changes.
For instance I remember that we first had

    DM-Upload-Allowed: Yes

in many control files.  It turned out that this was case sensitive and
so it was changed to

    DM-Upload-Allowed: yes

automatically in the whole repository.  This example might be weak because
you could argue that you could easily run your skript on two repositories
but experience shows that you tend to forget the other thing which is not
so interesting for whatever reason.  Also the Debian-Med developers page [1]
shows the current activity in our SVN.  If you have two repositories you have
to care for a mix of two repositories and there might be other tools invented
that gain an additional degree of complexity when having more than one
repository.

> The main disadvantage I can think of is that it is a nuisance for
> those who want to do project-wide work; for example, David Paleino
> once set up a watch file for insighttoolkit.  My response to that
> objection is that (at least in my experience with debian-med) the
> fraction of project-wide work is trivial and if I choose the minority
> vcs, I'll live with not having the attention.

This is also a disadvantage.

> The main ADvantage of allowing several vcs is inclusivity.  You well
> know that some refuse to use debian-med because it allows only SVN.  I
> hope debian-science won't repeat that mistake.

I agree that it is somewhat boring but I doubt that it is a problem of
the used VCS but rather a different kind of thinking which is connected
to the current discussion on debian-devel, how to store patches.  So
I rather think the reason for not using the Debian-Med VCS is the policy
to have all patches in debian/patches than the used VCS technique.

> Yes, I do appreciate beginner docs; I relied on them for learning svn
> and quilt.  I'm not adverse to learning git; but I will do it on my
> terms, not in order to join debian-science.

Sure - you are always free to choose.  That's the freedom we have as
volunteers.  If Debian Sciece is not attractive enough for you than
you maintain your packages as you did before.  The only chance to
draw you in is making Debian Science attractive enough that the
advantage for you (and others) is big enough.  I hope many people will
work on this. ;-)

Kind regards (and thanks for your input)

         Andreas.

[1] http://debian-med.alioth.debian.org/

--
http://fam-tille.de


--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Charles Plessy-12]

Le Tue, May 20, 2008 at 08:23:42AM +0200, Andreas Tille a écrit :

> For instance I remember that we first had
>
>    DM-Upload-Allowed: Yes
>
> in many control files.  It turned out that this was case sensitive and
> so it was changed to
>
>    DM-Upload-Allowed: yes
>
> automatically in the whole repository.  This example might be weak

Hi all,

another example:

I happily performed the Debian menu transition and the doc-base
transition on all the packages on our SVN, but would not have gone
through the hassle of doing a separate checkout for > 100 packages.

When FreeDesktop.org will have accepted a Science main category, I will
update all our .desktop files,…

Have a nice day,

--
Charles


--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Manuel Prinz-2]

Am Dienstag, den 20.05.2008, 15:44 +0900 schrieb Charles Plessy:

> Le Tue, May 20, 2008 at 08:23:42AM +0200, Andreas Tille a écrit :
> > For instance I remember that we first had
> >
> >    DM-Upload-Allowed: Yes
> >
> > in many control files.  It turned out that this was case sensitive and
> > so it was changed to
> >
> >    DM-Upload-Allowed: yes
> >
> > automatically in the whole repository.  This example might be weak
>
> Hi all,
>
> another example: [ Mass-update for menu and doc-base transition ]

To be clear right from the start: If we choose Git, mass-updates like
this are a little more hassle than in SVN, as you need to clone (aka
checkout) all packages locally, modify them and commit back. This is a
design decision and valid, I just wanted to mention it.

The good thing is: this can be handled by a tool such as the task tool I
wanted to write. (I actually started already.) It just needs the ability
to checkout all packages (trivial) and commit changes in all directories
(not so trivial but also not so hard, I guess). Anyway, I think that
mass-updates are a corner use-case: active maintainers will handle these
changes in a reasonable time frame and for all others the (yet to be
written) tools will hopefully assist us on mass-updates.

Best regards
Manuel

[David Bremner-4]

>>>>> "Sylvestre" == Sylvestre Ledru <sylvestre.ledru@...> writes:


    Sylvestre> If there is no strong objections, I will migrate the
    Sylvestre> current svn to git.

I have migrated sketch, vrr, and bibutils to git. Here a little script
I used.  Hopefully those new to git are not too horrified; one does
not need this kind of hackery in normal use.  The messy bit is to
transition from a package with only a /debian dir, to one with a full
upstream source compatible with git-buildpackage.

######################################################################
#!/bin/sh
package=bibutils
version=3.40
mkdir $package
cd $package
git-svn init --stdlayout --no-metadata svn://svn.debian.org/debian-science/$package
git-svn fetch
# drop upstream branch from svn
git-branch -d -r upstream

# create a new upstream branch based on recipe from
# http://upsilon.cc/~zack/blog/posts/2008/03/git-buildpackage_from_debian-only_to_debian+upstream/ 
#
git-symbolic-ref HEAD refs/heads/upstream
git rm --cached -r .
git commit --allow-empty -m 'initial upstream branch'
git checkout -f master
git merge upstream
git-import-orig --pristine-tar --no-dch ../tarballs/${package}_${version}.orig.tar.gz

######################################################################

from the parent directory, I ran

git clone --bare $package $package.git
rsync -cavz $package.git bremner-guest@...:/git/debian-science/packages

on alioth:

cd /git/debian/science/packages/$package.git
git --bare init --shared
chmod 755 hooks/post-update
git-update-server-info




--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Teemu Ikonen-2]

On Tue, May 20, 2008 at 12:54 PM, David Bremner <bremner@...> wrote:
> I have migrated sketch, vrr, and bibutils to git. Here a little script
> I used.

Does your script preserve revision history if it spans multiple
upstream releases?
If not, could they be somehow imported from snapshot.debian.net?

Just a suggestion, doing this could require a bit too much effort in
proportion to the benefits.

Teemu


--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[David Bremner-4]

>>>>> "Teemu" == Teemu Ikonen <tpikonen@...> writes:

    Teemu> On Tue, May 20, 2008 at 12:54 PM, David Bremner
    Teemu> <bremner@...> wrote:
    >> I have migrated sketch, vrr, and bibutils to git. Here a little
    >> script I used.

    Teemu> Does your script preserve revision history if it spans
    Teemu> multiple upstream releases?  If not, could they be somehow
    Teemu> imported from snapshot.debian.net?

Well, it preserves the edit history within /debian, which for me is
enough.  In some cases there is probably a clever way to pull the
upstream history from the svn repo using git-svn. For the packages
that were /debian only in the repo, that history seems not be there.
It should be possible to use git-import-orig to import tarballs in
sequence for all of the versions you want.

    Teemu> Just a suggestion, doing this could require a bit too much
    Teemu> effort in proportion to the benefits.

Well, it could be a nice general tool (sortof git-import-dsc++), but there
are many nice projects :-).

David




--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[Yaroslav Halchenko]

> Well, it preserves the edit history within /debian, which for me is
> enough.  In some cases there is probably a clever way to pull the
> upstream history from the svn repo using git-svn. For the packages
Just thought to share a piece of my experience although I doubt it would
generalize over any range of usecases ;)

I maintain fail2ban. Upstream sources used to be under CVS and then migrated
over to SVN. I kept my packaging in CVS and then in SVN (separate from
upstream). Then I decided to switch to git and at the end I ended up with
somewhat complicated beast (debcheckout fail2ban to see it), and it would fail
any collab standards out there,  but now the whole development (upstream) +
packaging (debian + released upstream sources) history is there, upstream's
SVN repository is linked via git-svn (thus I can cherry-pick needed commits to
fix bugs before upstream releases, and push/dcommit back into upstream
repository if needed).

Few obstacles I passed through on the way with might be worth mentioning and
sketching a solution:

* I switched to and them from the svn scheme to maintain only debian/, so some
upstream tags were pointing to bogus/empty directories. Thus I had to repopulate
upstream branch from the tarballs. I did it with smth like (upstream repository
tags them as X_Y_Z, but tarballs distributed are somewhat different, thus I
needed separate from original upstream branch, just for debian distributed
sources, it got taged upstream/X.Y.Z)

/bin/ls ../tarballs/fail2ban_0.*orig.tar.gz | \
 sed -e 's,.*fail.*_\(0.[.0-9]*\)\.orig.*,\1,g' |\
 sort | grep '^0' | \
  while read rev; do
    echo "Working on $rev"
    git merge --no-commit -s ours ${rev//./_}
        # remove everything (besides .git)
        rm * -rf
        # extract tarball
        tar -xzf ../tarballs/fail2ban_${rev}.orig.tar.gz
        mv fail2ban-0.*/* .
        rmdir fail2ban-0.*
        # add files
        git add `git ls-files -o`
        git commit -m "Upgraded to fresh upstream ${rev}" -a
        git tag upstream/${rev}
    git status
  done

* git-svn pulls in tags from svn as branches, so you would need to tag
appropriately and remove those bogus branches for the sake of sanity  

git branch -r | sed -rne 's, *upstream-repo/tags/FAIL2BAN-([^@]+)$,\1,p' | \
while read tag; do
git tag $tag upstream-repo/tags/FAIL2BAN-${tag}^ && \
  git branch -r -d upstream-repo/tags/FAIL2BAN-$tag;
done

--
Yaroslav Halchenko
Research Assistant, Psychology Department, Rutgers-Newark
Student  Ph.D. @ CS Dept. NJIT
Office: [phone]x263 | FWD: 82823 | Fax: [phone]
        101 Warren Str, Smith Hall, Rm 4-105, Newark NJ 07102
WWW:     http://www.linkedin.com/in/yarik       


--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...

[David Bremner-4]

>>>>> "David" == David Bremner <bremner@...> writes:

    David> I have migrated sketch, vrr, and bibutils to git. Here a
    David> little script I used.  

By the way, if anyone was trying to use my recipe, I missed having the
--author-file option to svn. This meant the author and email were
"bremner-guest <unhelpful hexstring>".  A way to fix this in
postprocessing is given in

     http://www.cs.unb.ca/~bremner/blog/posts/svn_to_git/

But as always, it is better to do things right the first time.

David

P.S. If anybody has one of my packages checked out, now you probably
can't push changes back. Sorry. email me a patch...


--
To UNSUBSCRIBE, email to debian-science-request@...
with a subject of "unsubscribe". Trouble? Contact listmaster@...