Status: 22-Mar-2010 - Git organization

classic Classic list List threaded Threaded
16 messages Options
Reply | Threaded
Open this post in threaded view
|

Status: 22-Mar-2010 - Git organization

weierophinney
Administrator
Greetings, all --

Based on the wiki and the poll on this list, it looks like it makes
sense to move forward with git. We're still trying to iron on some
details about how this will work, but some details are falling into
place.

First, the things we plan:

 * Git will be hosted on the ZF servers. This is so that we can have
   full control over hooks (i.e., no network latency from web hooks),
   integrate with other tools (e.g. viewgit, JIRA, etc.), and more. We
   will offer both anonymous access as well as selective writable
   access.

   Some will say, "why not just use Github?" The above does not preclude
   using github in any way; git allows you to specify multiple remotes
   from your repository. I already plan to have a mirror of my personal
   repo on github, as well as push to the ZF repo from the same
   location. This will allow people to use some of githubs powerful
   features such as pull queues while still allowing us to retain some
   control over the repository. (It also insulates us from githubs
   growing pains a bit -- I love github, but they are certainly
   suffering from rapid growth the last few months!)

 * We plan to have a read-only subversion mirror of the git repository.
   This will allow those using svn:externals to continue to do so, as
   well as allow folks more familiar with svn to continue to provide
   patches.

Next, things we're still researching:

 * How to handle contributions. Since we require a signed CLA, we need
   some way of verifying that a patch came from an author who has signed
   the CLA. There have been some suggestions that this may be possible
   with some sort of pre-commit verifications when committing to the
   master repository (or when someone with commit privileges commits to
   their own repository). We're still researching how this might be
   implemented.

 * Submodules...
   For those unfamiliar with git submodules, they are roughly equivalent
   to svn:externals. I say "roughly", because they differ in many ways:
   they have to be manually initialized on each person's repository
   (though they can be defined in the master repository); they need to
   be manually updated periodically (unlike svn:externals, where anytime
   you do an "svn up", it also syncs the external); etc.

   They are quite powerful, however, and give us the ability to offer a
   "lean" initial checkout, allowing developers to sync in only the
   submodules they plan to utilize or commit to.

   On this note, I've fielded a number of suggestions. Some were
   suggesting the library directory be its own submodule, some suggested
   all top-level directories be their own submodule, etc. I've done some
   experimentation, and now propose the following structure:

    MAIN REPO:
        INSTALL.txt
        LICENSE.txt
        README.txt
        bin/
        library
        tests/
        documentation/manual/
            Makefile
            README.txt
            en/
    SUBMODULES:
        demos/
        resources/
        tools/
        documentation/manual/:
            all languages except "en"

   Now, for an explanation of the "whys" behind this. As contributors,
   it's important to make atomic commits: your code fix should also
   include a corresponding unit test, and potentially a documentation
   commit as well. If we split the library and tests directories out
   into their own submodules, we cannot do so; the code and test must be
   committed separately, which is (a) more work, and (b) unintuitive.
   The same goes for related documentation commits.

   Translations, on the other hand, are usually managed in a
   per-language capacity, flowing from English to the target language.
   Most developers do not need the docbook source for anything but their
   target language and English (the latter because it's used for
   fallbacks, and because it contains the build scripts). While we
   potentially lose the history of an "svn cp", I think this
   organization provides reasonable benefits.

   Finally, the demos, resources, and tools (the latter is a new
   directory) are all "opt-in" items -- if you need them, you can pull
   them in; if you don't, there's no reason to.

I'd love to get some feedback on the incomplete items. I think the
submodule layout I've come up with makes the most sense for contributors
(and based on testing, it's the cleanest and simplest way I've found to
accomplish most common development tasks on our tree), but am open to
further suggestions. If you have any ideas on handling ACLs, I'd love to
hear them, and sooner rather than later!

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

Simone Carletti
Hello Matthew,

I don't know the details about how you are going to manage the git push/pull task from a local machine to a git server, however I suggest you to have a look at the following document.

It describes how gitosis works and, AFAIK, it roughly resemble to the current GitHub and Gitorious system.
Basically, you have one unique unix user which handles all the communication from the local machine to the master repository.
All users pull and push changes using this user. Internally the system checks the public ssh key of the committer agains a whitelist which is stored itself in a Git repos.

In your case, the whitelist can be the list of all users which signed the CLA and are allowed to send commits to the repository.

I hope this can help you,
-- Simone


On Mon, Mar 22, 2010 at 1:36 PM, Matthew Weier O'Phinney <[hidden email]> wrote:
 If you have any ideas on handling ACLs, I'd love to
hear them, and sooner rather than later!

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc



--
Simone Carletti

Site & Blog: http://www.simonecarletti.com
Email: [hidden email]
LinkedIn: http://linkedin.com/in/weppos
Nick: weppos | Skype: weppos
Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

Josh Butts
In reply to this post by weierophinney
Matthew,

Just as an FYI, the latest version of Fisheye does support Git (you mentioned viewgit below). We're using it at my company and have had no problems with it.

-Josh

On Mar 22, 2010, at 7:36 AM, Matthew Weier O'Phinney wrote:

> Greetings, all --
>
> Based on the wiki and the poll on this list, it looks like it makes
> sense to move forward with git. We're still trying to iron on some
> details about how this will work, but some details are falling into
> place.
>
> First, the things we plan:
>
> * Git will be hosted on the ZF servers. This is so that we can have
>   full control over hooks (i.e., no network latency from web hooks),
>   integrate with other tools (e.g. viewgit, JIRA, etc.), and more. We
>   will offer both anonymous access as well as selective writable
>   access.
>
>   Some will say, "why not just use Github?" The above does not preclude
>   using github in any way; git allows you to specify multiple remotes
>   from your repository. I already plan to have a mirror of my personal
>   repo on github, as well as push to the ZF repo from the same
>   location. This will allow people to use some of githubs powerful
>   features such as pull queues while still allowing us to retain some
>   control over the repository. (It also insulates us from githubs
>   growing pains a bit -- I love github, but they are certainly
>   suffering from rapid growth the last few months!)
>
> * We plan to have a read-only subversion mirror of the git repository.
>   This will allow those using svn:externals to continue to do so, as
>   well as allow folks more familiar with svn to continue to provide
>   patches.
>
> Next, things we're still researching:
>
> * How to handle contributions. Since we require a signed CLA, we need
>   some way of verifying that a patch came from an author who has signed
>   the CLA. There have been some suggestions that this may be possible
>   with some sort of pre-commit verifications when committing to the
>   master repository (or when someone with commit privileges commits to
>   their own repository). We're still researching how this might be
>   implemented.
>
> * Submodules...
>   For those unfamiliar with git submodules, they are roughly equivalent
>   to svn:externals. I say "roughly", because they differ in many ways:
>   they have to be manually initialized on each person's repository
>   (though they can be defined in the master repository); they need to
>   be manually updated periodically (unlike svn:externals, where anytime
>   you do an "svn up", it also syncs the external); etc.
>
>   They are quite powerful, however, and give us the ability to offer a
>   "lean" initial checkout, allowing developers to sync in only the
>   submodules they plan to utilize or commit to.
>
>   On this note, I've fielded a number of suggestions. Some were
>   suggesting the library directory be its own submodule, some suggested
>   all top-level directories be their own submodule, etc. I've done some
>   experimentation, and now propose the following structure:
>
>    MAIN REPO:
>        INSTALL.txt
>        LICENSE.txt
>        README.txt
>        bin/
>        library
>        tests/
>        documentation/manual/
>            Makefile
>            README.txt
>            en/
>    SUBMODULES:
>        demos/
>        resources/
>        tools/
>        documentation/manual/:
>            all languages except "en"
>
>   Now, for an explanation of the "whys" behind this. As contributors,
>   it's important to make atomic commits: your code fix should also
>   include a corresponding unit test, and potentially a documentation
>   commit as well. If we split the library and tests directories out
>   into their own submodules, we cannot do so; the code and test must be
>   committed separately, which is (a) more work, and (b) unintuitive.
>   The same goes for related documentation commits.
>
>   Translations, on the other hand, are usually managed in a
>   per-language capacity, flowing from English to the target language.
>   Most developers do not need the docbook source for anything but their
>   target language and English (the latter because it's used for
>   fallbacks, and because it contains the build scripts). While we
>   potentially lose the history of an "svn cp", I think this
>   organization provides reasonable benefits.
>
>   Finally, the demos, resources, and tools (the latter is a new
>   directory) are all "opt-in" items -- if you need them, you can pull
>   them in; if you don't, there's no reason to.
>
> I'd love to get some feedback on the incomplete items. I think the
> submodule layout I've come up with makes the most sense for contributors
> (and based on testing, it's the cleanest and simplest way I've found to
> accomplish most common development tasks on our tree), but am open to
> further suggestions. If you have any ideas on handling ACLs, I'd love to
> hear them, and sooner rather than later!
>
> --
> Matthew Weier O'Phinney
> Project Lead            | [hidden email]
> Zend Framework          | http://framework.zend.com/
> PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

Benjamin Eberlei-2
In reply to this post by weierophinney

We were planning to move Doctrine 2 development to Git also, however
one problem we saw was the alpha/beta-only git support for Jira, have
you evaluated the possibilities in this area?

On Mon, 22 Mar 2010 08:36:52 -0400, Matthew Weier O'Phinney
<[hidden email]> wrote:

> Greetings, all --
>
> Based on the wiki and the poll on this list, it looks like it makes
> sense to move forward with git. We're still trying to iron on some
> details about how this will work, but some details are falling into
> place.
>
> First, the things we plan:
>
>  * Git will be hosted on the ZF servers. This is so that we can have
>    full control over hooks (i.e., no network latency from web hooks),
>    integrate with other tools (e.g. viewgit, JIRA, etc.), and more. We
>    will offer both anonymous access as well as selective writable
>    access.
>
>    Some will say, "why not just use Github?" The above does not preclude
>    using github in any way; git allows you to specify multiple remotes
>    from your repository. I already plan to have a mirror of my personal
>    repo on github, as well as push to the ZF repo from the same
>    location. This will allow people to use some of githubs powerful
>    features such as pull queues while still allowing us to retain some
>    control over the repository. (It also insulates us from githubs
>    growing pains a bit -- I love github, but they are certainly
>    suffering from rapid growth the last few months!)
>
>  * We plan to have a read-only subversion mirror of the git repository.
>    This will allow those using svn:externals to continue to do so, as
>    well as allow folks more familiar with svn to continue to provide
>    patches.
>
> Next, things we're still researching:
>
>  * How to handle contributions. Since we require a signed CLA, we need
>    some way of verifying that a patch came from an author who has signed
>    the CLA. There have been some suggestions that this may be possible
>    with some sort of pre-commit verifications when committing to the
>    master repository (or when someone with commit privileges commits to
>    their own repository). We're still researching how this might be
>    implemented.
>
>  * Submodules...
>    For those unfamiliar with git submodules, they are roughly equivalent
>    to svn:externals. I say "roughly", because they differ in many ways:
>    they have to be manually initialized on each person's repository
>    (though they can be defined in the master repository); they need to
>    be manually updated periodically (unlike svn:externals, where anytime
>    you do an "svn up", it also syncs the external); etc.
>
>    They are quite powerful, however, and give us the ability to offer a
>    "lean" initial checkout, allowing developers to sync in only the
>    submodules they plan to utilize or commit to.
>
>    On this note, I've fielded a number of suggestions. Some were
>    suggesting the library directory be its own submodule, some suggested
>    all top-level directories be their own submodule, etc. I've done some
>    experimentation, and now propose the following structure:
>
>     MAIN REPO:
>         INSTALL.txt
>         LICENSE.txt
>         README.txt
>         bin/
>         library
>         tests/
>         documentation/manual/
>             Makefile
>             README.txt
>             en/
>     SUBMODULES:
>         demos/
>         resources/
>         tools/
>         documentation/manual/:
>             all languages except "en"
>
>    Now, for an explanation of the "whys" behind this. As contributors,
>    it's important to make atomic commits: your code fix should also
>    include a corresponding unit test, and potentially a documentation
>    commit as well. If we split the library and tests directories out
>    into their own submodules, we cannot do so; the code and test must be
>    committed separately, which is (a) more work, and (b) unintuitive.
>    The same goes for related documentation commits.
>
>    Translations, on the other hand, are usually managed in a
>    per-language capacity, flowing from English to the target language.
>    Most developers do not need the docbook source for anything but their
>    target language and English (the latter because it's used for
>    fallbacks, and because it contains the build scripts). While we
>    potentially lose the history of an "svn cp", I think this
>    organization provides reasonable benefits.
>
>    Finally, the demos, resources, and tools (the latter is a new
>    directory) are all "opt-in" items -- if you need them, you can pull
>    them in; if you don't, there's no reason to.
>
> I'd love to get some feedback on the incomplete items. I think the
> submodule layout I've come up with makes the most sense for contributors
> (and based on testing, it's the cleanest and simplest way I've found to
> accomplish most common development tasks on our tree), but am open to
> further suggestions. If you have any ideas on handling ACLs, I'd love to
> hear them, and sooner rather than later!

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

weierophinney
Administrator
In reply to this post by Simone Carletti
-- Simone Carletti <[hidden email]> wrote
(on Monday, 22 March 2010, 01:46 PM +0100):
> I don't know the details about how you are going to manage the git push/pull
> task from a local machine to a git server, however I suggest you to have a look
> at the following document.
> http://scie.nti.st/2007/11/14/hosting-git-repositories-the-easy-and-secure-way
>
> It describes how gitosis works and, AFAIK, it roughly resemble to the current
> GitHub and Gitorious system.

Indeed, I've been playing with Gitorious and git-daemon (former for
write-access, latter for read-only; they work together quite well).

> Basically, you have one unique unix user which handles all the communication
> from the local machine to the master repository.
> All users pull and push changes using this user. Internally the system checks
> the public ssh key of the committer agains a whitelist which is stored itself
> in a Git repos.
>
> In your case, the whitelist can be the list of all users which signed the CLA
> and are allowed to send commits to the repository.

THe problem is going to be ensuring that patches that the contributors
have merged and commit fall under the CLA. However, if they are doing
true merges, we can likely at the least look at the email associated
with each merged commit (I've seen some hooks for doing this exact
thing). This could potentially be spoofed, but so can svn credentials,
to be honest.

> I hope this can help you,
> -- Simone
>
>
> On Mon, Mar 22, 2010 at 1:36 PM, Matthew Weier O'Phinney <[hidden email]>
> wrote:
>
>      If you have any ideas on handling ACLs, I'd love to
>     hear them, and sooner rather than later!
>
>     --
>     Matthew Weier O'Phinney
>     Project Lead            | [hidden email]
>     Zend Framework          | http://framework.zend.com/
>     PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
>
>
>
>
> --
> Simone Carletti
>
> Site & Blog: http://www.simonecarletti.com
> Email: [hidden email]
> LinkedIn: http://linkedin.com/in/weppos
> Nick: weppos | Skype: weppos

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

weierophinney
Administrator
In reply to this post by Josh Butts
-- Josh Butts <[hidden email]> wrote
(on Monday, 22 March 2010, 08:07 AM -0500):
> Matthew,
>
> Just as an FYI, the latest version of Fisheye does support Git (you
> mentioned viewgit below). We're using it at my company and have had no
> problems with it.

Yeah - I heard that.

We're planning on moving from basically all the Atlassian tools except
JIRA. The reasons are:

 * Lack of familiarity with Java, which makes maintaining/debugging
   resource intensive. While we can get support from Atlassian and/or
   their partners, the problem is that we typically need a _fast_
   solution and/or one that works within the confines of our existing
   server infrastructure.

   If we can move to PHP-based tools, we can manage them much easier
   with our existing resources.

 * Ability to customize and embed tools in our site. The Atlassian tools
   have minimal customization capabilities, and as such stick out as not
   being part of the ZF site. We'd really like to have them look more
   like the rest of the site, to make the experience more cohesive and
   less jarring when you perform different tasks. The PHP apps are more
   conducive to this -- not just by design, but because we can easily
   manipulate the source code.

 * Performance. Our site performance often lags due to the architecture
   we've had to create to support having the Atlassian tools in place.
   The fewer tools we have up, the better the site will perform.

I've tinkered with viewgit quite a bit, and I think it will satisfy the
above situations quite well.

As for JIRA, it's really a quite good issue tracker, and I've heard
anecdotally that it's integration with git repositories works quite
well. As such, I see no reason to remove it from our stack.


> On Mar 22, 2010, at 7:36 AM, Matthew Weier O'Phinney wrote:
>
> > Greetings, all --
> >
> > Based on the wiki and the poll on this list, it looks like it makes
> > sense to move forward with git. We're still trying to iron on some
> > details about how this will work, but some details are falling into
> > place.
> >
> > First, the things we plan:
> >
> > * Git will be hosted on the ZF servers. This is so that we can have
> >   full control over hooks (i.e., no network latency from web hooks),
> >   integrate with other tools (e.g. viewgit, JIRA, etc.), and more. We
> >   will offer both anonymous access as well as selective writable
> >   access.
> >
> >   Some will say, "why not just use Github?" The above does not preclude
> >   using github in any way; git allows you to specify multiple remotes
> >   from your repository. I already plan to have a mirror of my personal
> >   repo on github, as well as push to the ZF repo from the same
> >   location. This will allow people to use some of githubs powerful
> >   features such as pull queues while still allowing us to retain some
> >   control over the repository. (It also insulates us from githubs
> >   growing pains a bit -- I love github, but they are certainly
> >   suffering from rapid growth the last few months!)
> >
> > * We plan to have a read-only subversion mirror of the git repository.
> >   This will allow those using svn:externals to continue to do so, as
> >   well as allow folks more familiar with svn to continue to provide
> >   patches.
> >
> > Next, things we're still researching:
> >
> > * How to handle contributions. Since we require a signed CLA, we need
> >   some way of verifying that a patch came from an author who has signed
> >   the CLA. There have been some suggestions that this may be possible
> >   with some sort of pre-commit verifications when committing to the
> >   master repository (or when someone with commit privileges commits to
> >   their own repository). We're still researching how this might be
> >   implemented.
> >
> > * Submodules...
> >   For those unfamiliar with git submodules, they are roughly equivalent
> >   to svn:externals. I say "roughly", because they differ in many ways:
> >   they have to be manually initialized on each person's repository
> >   (though they can be defined in the master repository); they need to
> >   be manually updated periodically (unlike svn:externals, where anytime
> >   you do an "svn up", it also syncs the external); etc.
> >
> >   They are quite powerful, however, and give us the ability to offer a
> >   "lean" initial checkout, allowing developers to sync in only the
> >   submodules they plan to utilize or commit to.
> >
> >   On this note, I've fielded a number of suggestions. Some were
> >   suggesting the library directory be its own submodule, some suggested
> >   all top-level directories be their own submodule, etc. I've done some
> >   experimentation, and now propose the following structure:
> >
> >    MAIN REPO:
> >        INSTALL.txt
> >        LICENSE.txt
> >        README.txt
> >        bin/
> >        library
> >        tests/
> >        documentation/manual/
> >            Makefile
> >            README.txt
> >            en/
> >    SUBMODULES:
> >        demos/
> >        resources/
> >        tools/
> >        documentation/manual/:
> >            all languages except "en"
> >
> >   Now, for an explanation of the "whys" behind this. As contributors,
> >   it's important to make atomic commits: your code fix should also
> >   include a corresponding unit test, and potentially a documentation
> >   commit as well. If we split the library and tests directories out
> >   into their own submodules, we cannot do so; the code and test must be
> >   committed separately, which is (a) more work, and (b) unintuitive.
> >   The same goes for related documentation commits.
> >
> >   Translations, on the other hand, are usually managed in a
> >   per-language capacity, flowing from English to the target language.
> >   Most developers do not need the docbook source for anything but their
> >   target language and English (the latter because it's used for
> >   fallbacks, and because it contains the build scripts). While we
> >   potentially lose the history of an "svn cp", I think this
> >   organization provides reasonable benefits.
> >
> >   Finally, the demos, resources, and tools (the latter is a new
> >   directory) are all "opt-in" items -- if you need them, you can pull
> >   them in; if you don't, there's no reason to.
> >
> > I'd love to get some feedback on the incomplete items. I think the
> > submodule layout I've come up with makes the most sense for contributors
> > (and based on testing, it's the cleanest and simplest way I've found to
> > accomplish most common development tasks on our tree), but am open to
> > further suggestions. If you have any ideas on handling ACLs, I'd love to
> > hear them, and sooner rather than later!
> >
> > --
> > Matthew Weier O'Phinney
> > Project Lead            | [hidden email]
> > Zend Framework          | http://framework.zend.com/
> > PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
>

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

Eric Clemmons
In reply to this post by weierophinney
So, would a hypothetical (Github) workflow be as follows?

1.  Fork Matthew's ZF repo
2.  Create local branch for adding feature (ie, OAuth2.0 support)
3.  Make atomic commits for unit tests, code changes, and new documentation
4.  Push branch to personal repo
5.  Issue a pull request for said feature
6.  Matthew runs a script that verifies my Github username against the CLAs, subsequently adds the remote and pulls the feature branch.

Sounds like you'll be really busy, Matthew.

I'm sure the majority of contributors would not have direct write access to the Zend repo, which means the rest of the requests will have to be pull requests and properly QA'd prior to merging.

-Eric

On Mar 22, 2010, at 7:36 AM, Matthew Weier O'Phinney wrote:

> Greetings, all --
>
> Based on the wiki and the poll on this list, it looks like it makes
> sense to move forward with git. We're still trying to iron on some
> details about how this will work, but some details are falling into
> place.
>
> First, the things we plan:
>
> * Git will be hosted on the ZF servers. This is so that we can have
>   full control over hooks (i.e., no network latency from web hooks),
>   integrate with other tools (e.g. viewgit, JIRA, etc.), and more. We
>   will offer both anonymous access as well as selective writable
>   access.
>
>   Some will say, "why not just use Github?" The above does not preclude
>   using github in any way; git allows you to specify multiple remotes
>   from your repository. I already plan to have a mirror of my personal
>   repo on github, as well as push to the ZF repo from the same
>   location. This will allow people to use some of githubs powerful
>   features such as pull queues while still allowing us to retain some
>   control over the repository. (It also insulates us from githubs
>   growing pains a bit -- I love github, but they are certainly
>   suffering from rapid growth the last few months!)
>
> * We plan to have a read-only subversion mirror of the git repository.
>   This will allow those using svn:externals to continue to do so, as
>   well as allow folks more familiar with svn to continue to provide
>   patches.
>
> Next, things we're still researching:
>
> * How to handle contributions. Since we require a signed CLA, we need
>   some way of verifying that a patch came from an author who has signed
>   the CLA. There have been some suggestions that this may be possible
>   with some sort of pre-commit verifications when committing to the
>   master repository (or when someone with commit privileges commits to
>   their own repository). We're still researching how this might be
>   implemented.
>
> * Submodules...
>   For those unfamiliar with git submodules, they are roughly equivalent
>   to svn:externals. I say "roughly", because they differ in many ways:
>   they have to be manually initialized on each person's repository
>   (though they can be defined in the master repository); they need to
>   be manually updated periodically (unlike svn:externals, where anytime
>   you do an "svn up", it also syncs the external); etc.
>
>   They are quite powerful, however, and give us the ability to offer a
>   "lean" initial checkout, allowing developers to sync in only the
>   submodules they plan to utilize or commit to.
>
>   On this note, I've fielded a number of suggestions. Some were
>   suggesting the library directory be its own submodule, some suggested
>   all top-level directories be their own submodule, etc. I've done some
>   experimentation, and now propose the following structure:
>
>    MAIN REPO:
>        INSTALL.txt
>        LICENSE.txt
>        README.txt
>        bin/
>        library
>        tests/
>        documentation/manual/
>            Makefile
>            README.txt
>            en/
>    SUBMODULES:
>        demos/
>        resources/
>        tools/
>        documentation/manual/:
>            all languages except "en"
>
>   Now, for an explanation of the "whys" behind this. As contributors,
>   it's important to make atomic commits: your code fix should also
>   include a corresponding unit test, and potentially a documentation
>   commit as well. If we split the library and tests directories out
>   into their own submodules, we cannot do so; the code and test must be
>   committed separately, which is (a) more work, and (b) unintuitive.
>   The same goes for related documentation commits.
>
>   Translations, on the other hand, are usually managed in a
>   per-language capacity, flowing from English to the target language.
>   Most developers do not need the docbook source for anything but their
>   target language and English (the latter because it's used for
>   fallbacks, and because it contains the build scripts). While we
>   potentially lose the history of an "svn cp", I think this
>   organization provides reasonable benefits.
>
>   Finally, the demos, resources, and tools (the latter is a new
>   directory) are all "opt-in" items -- if you need them, you can pull
>   them in; if you don't, there's no reason to.
>
> I'd love to get some feedback on the incomplete items. I think the
> submodule layout I've come up with makes the most sense for contributors
> (and based on testing, it's the cleanest and simplest way I've found to
> accomplish most common development tasks on our tree), but am open to
> further suggestions. If you have any ideas on handling ACLs, I'd love to
> hear them, and sooner rather than later!
>
> --
> Matthew Weier O'Phinney
> Project Lead            | [hidden email]
> Zend Framework          | http://framework.zend.com/
> PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

Andy Fowler
Eric, that sounds right to me -- it's how most Github workflows operate, and it'd be terrific if we could use Github to submit pull requests to Matthew, even though he'll be merging them into the Zend-hosted repo.

A quick note on "atomic commits" in git. One of the things I had to learn when moving to Git was to get away from huge end-of-feature commits, like one would make in SVN. Eventually, it becomes easier to commit, revert, commit, revert many times during the course of a feature. At the end of the feature branch you can squash those many commits into a single atomic commit that contains all unit tests and docs:

http://reinh.com/blog/2009/03/02/a-git-workflow-for-agile-teams.html

As a CLA'd but rare contributor, I'm really looking forward to the move to Git. In the past few months, I had given up on submitting patches. Nagging the IRC channel to pull something from JIRA was terribly inefficient -- hopefully this can smooth things along!

Andy


On Mar 22, 2010, at 11:21 AM, Eric Clemmons wrote:

> So, would a hypothetical (Github) workflow be as follows?
> ...
> 3.  Make atomic commits for unit tests, code changes, and new documentation
> ...
> On Mar 22, 2010, at 7:36 AM, Matthew Weier O'Phinney wrote:
>
>> ...
>>  Now, for an explanation of the "whys" behind this. As contributors,
>>  it's important to make atomic commits: your code fix should also
>>  include a corresponding unit test, and potentially a documentation
>>  commit as well. If we split the library and tests directories out
>>  into their own submodules, we cannot do so; the code and test must be
>>  committed separately, which is (a) more work, and (b) unintuitive.
>>  The same goes for related documentation commits.

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

weierophinney
Administrator
In reply to this post by Eric Clemmons
-- Eric Clemmons <[hidden email]> wrote
(on Monday, 22 March 2010, 09:45 AM -0500):

> So, would a hypothetical (Github) workflow be as follows?
>
> 1.  Fork Matthew's ZF repo
> 2.  Create local branch for adding feature (ie, OAuth2.0 support)
> 3.  Make atomic commits for unit tests, code changes, and new documentation
> 4.  Push branch to personal repo
> 5.  Issue a pull request for said feature
> 6.  Matthew runs a script that verifies my Github username against the
>     CLAs, subsequently adds the remote and pulls the feature branch.
>
> Sounds like you'll be really busy, Matthew.
>
> I'm sure the majority of contributors would not have direct write
> access to the Zend repo, which means the rest of the requests will
> have to be pull requests and properly QA'd prior to merging.

(replied personally earlier, publicly here)

I wouldn't be the only person with commit access to the repository, for
the very reason you describe. We will either give access to any active
component maintainers, create a QA team, or appoint community release
managers who would perform the pull requests.

In other words, there are a variety of approaches that can solve the
bottlenecks, and we'll evaluate which ones work best for the community.


> On Mar 22, 2010, at 7:36 AM, Matthew Weier O'Phinney wrote:
>
> > Greetings, all --
> >
> > Based on the wiki and the poll on this list, it looks like it makes
> > sense to move forward with git. We're still trying to iron on some
> > details about how this will work, but some details are falling into
> > place.
> >
> > First, the things we plan:
> >
> > * Git will be hosted on the ZF servers. This is so that we can have
> >   full control over hooks (i.e., no network latency from web hooks),
> >   integrate with other tools (e.g. viewgit, JIRA, etc.), and more. We
> >   will offer both anonymous access as well as selective writable
> >   access.
> >
> >   Some will say, "why not just use Github?" The above does not preclude
> >   using github in any way; git allows you to specify multiple remotes
> >   from your repository. I already plan to have a mirror of my personal
> >   repo on github, as well as push to the ZF repo from the same
> >   location. This will allow people to use some of githubs powerful
> >   features such as pull queues while still allowing us to retain some
> >   control over the repository. (It also insulates us from githubs
> >   growing pains a bit -- I love github, but they are certainly
> >   suffering from rapid growth the last few months!)
> >
> > * We plan to have a read-only subversion mirror of the git repository.
> >   This will allow those using svn:externals to continue to do so, as
> >   well as allow folks more familiar with svn to continue to provide
> >   patches.
> >
> > Next, things we're still researching:
> >
> > * How to handle contributions. Since we require a signed CLA, we need
> >   some way of verifying that a patch came from an author who has signed
> >   the CLA. There have been some suggestions that this may be possible
> >   with some sort of pre-commit verifications when committing to the
> >   master repository (or when someone with commit privileges commits to
> >   their own repository). We're still researching how this might be
> >   implemented.
> >
> > * Submodules...
> >   For those unfamiliar with git submodules, they are roughly equivalent
> >   to svn:externals. I say "roughly", because they differ in many ways:
> >   they have to be manually initialized on each person's repository
> >   (though they can be defined in the master repository); they need to
> >   be manually updated periodically (unlike svn:externals, where anytime
> >   you do an "svn up", it also syncs the external); etc.
> >
> >   They are quite powerful, however, and give us the ability to offer a
> >   "lean" initial checkout, allowing developers to sync in only the
> >   submodules they plan to utilize or commit to.
> >
> >   On this note, I've fielded a number of suggestions. Some were
> >   suggesting the library directory be its own submodule, some suggested
> >   all top-level directories be their own submodule, etc. I've done some
> >   experimentation, and now propose the following structure:
> >
> >    MAIN REPO:
> >        INSTALL.txt
> >        LICENSE.txt
> >        README.txt
> >        bin/
> >        library
> >        tests/
> >        documentation/manual/
> >            Makefile
> >            README.txt
> >            en/
> >    SUBMODULES:
> >        demos/
> >        resources/
> >        tools/
> >        documentation/manual/:
> >            all languages except "en"
> >
> >   Now, for an explanation of the "whys" behind this. As contributors,
> >   it's important to make atomic commits: your code fix should also
> >   include a corresponding unit test, and potentially a documentation
> >   commit as well. If we split the library and tests directories out
> >   into their own submodules, we cannot do so; the code and test must be
> >   committed separately, which is (a) more work, and (b) unintuitive.
> >   The same goes for related documentation commits.
> >
> >   Translations, on the other hand, are usually managed in a
> >   per-language capacity, flowing from English to the target language.
> >   Most developers do not need the docbook source for anything but their
> >   target language and English (the latter because it's used for
> >   fallbacks, and because it contains the build scripts). While we
> >   potentially lose the history of an "svn cp", I think this
> >   organization provides reasonable benefits.
> >
> >   Finally, the demos, resources, and tools (the latter is a new
> >   directory) are all "opt-in" items -- if you need them, you can pull
> >   them in; if you don't, there's no reason to.
> >
> > I'd love to get some feedback on the incomplete items. I think the
> > submodule layout I've come up with makes the most sense for contributors
> > (and based on testing, it's the cleanest and simplest way I've found to
> > accomplish most common development tasks on our tree), but am open to
> > further suggestions. If you have any ideas on handling ACLs, I'd love to
> > hear them, and sooner rather than later!

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

weierophinney
Administrator
In reply to this post by Andy Fowler
-- Andy Fowler <[hidden email]> wrote
(on Monday, 22 March 2010, 11:38 AM -0400):
> Eric, that sounds right to me -- it's how most Github workflows
> operate, and it'd be terrific if we could use Github to submit pull
> requests to Matthew, even though he'll be merging them into the
> Zend-hosted repo.

Please see my previous response to Eric.

> A quick note on "atomic commits" in git. One of the things I had to
> learn when moving to Git was to get away from huge end-of-feature
> commits, like one would make in SVN. Eventually, it becomes easier to
> commit, revert, commit, revert many times during the course of a
> feature. At the end of the feature branch you can squash those many
> commits into a single atomic commit that contains all unit tests and
> docs:

Yes, but that's exactly what I was getting at. If we do library/ and
tests/ and documentation/ all as submodules, we can't have that final
atomic commit that has the tests and docs and code all together. :)

I thoroughly enjoy the flexibility Git offers, but when it comes to
the master repository, we need the atomicity -- and the proposed
submodule layout I provided earlier in the thread answers that.

> http://reinh.com/blog/2009/03/02/a-git-workflow-for-agile-teams.html
>
> As a CLA'd but rare contributor, I'm really looking forward to the
> move to Git. In the past few months, I had given up on submitting
> patches. Nagging the IRC channel to pull something from JIRA was
> terribly inefficient -- hopefully this can smooth things along!

That's my hope as well.


> On Mar 22, 2010, at 11:21 AM, Eric Clemmons wrote:
>
> > So, would a hypothetical (Github) workflow be as follows?
> > ...
> > 3.  Make atomic commits for unit tests, code changes, and new documentation
> > ...
> > On Mar 22, 2010, at 7:36 AM, Matthew Weier O'Phinney wrote:
> >
> >> ...
> >>  Now, for an explanation of the "whys" behind this. As contributors,
> >>  it's important to make atomic commits: your code fix should also
> >>  include a corresponding unit test, and potentially a documentation
> >>  commit as well. If we split the library and tests directories out
> >>  into their own submodules, we cannot do so; the code and test must be
> >>  committed separately, which is (a) more work, and (b) unintuitive.
> >>  The same goes for related documentation commits.
>

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

Pieter Kokx
Matthew Weier O'Phinney schreef:
-- Andy Fowler [hidden email] wrote
(on Monday, 22 March 2010, 11:38 AM -0400):
  
Eric, that sounds right to me -- it's how most Github workflows
operate, and it'd be terrific if we could use Github to submit pull
requests to Matthew, even though he'll be merging them into the
Zend-hosted repo.
    

Please see my previous response to Eric.

  
A quick note on "atomic commits" in git. One of the things I had to
learn when moving to Git was to get away from huge end-of-feature
commits, like one would make in SVN. Eventually, it becomes easier to
commit, revert, commit, revert many times during the course of a
feature. At the end of the feature branch you can squash those many
commits into a single atomic commit that contains all unit tests and
docs:
    

Yes, but that's exactly what I was getting at. If we do library/ and
tests/ and documentation/ all as submodules, we can't have that final
atomic commit that has the tests and docs and code all together. :) 
  
Well, actually you can. By making a commit to the super repository that includes the new commits to the submodules, you have that atomic commit that combines tests, code and docs.
I thoroughly enjoy the flexibility Git offers, but when it comes to
the master repository, we need the atomicity -- and the proposed
submodule layout I provided earlier in the thread answers that.

  
http://reinh.com/blog/2009/03/02/a-git-workflow-for-agile-teams.html

As a CLA'd but rare contributor, I'm really looking forward to the
move to Git. In the past few months, I had given up on submitting
patches. Nagging the IRC channel to pull something from JIRA was
terribly inefficient -- hopefully this can smooth things along!
    

That's my hope as well.


  
On Mar 22, 2010, at 11:21 AM, Eric Clemmons wrote:

    
So, would a hypothetical (Github) workflow be as follows?
...
3.  Make atomic commits for unit tests, code changes, and new documentation
...
On Mar 22, 2010, at 7:36 AM, Matthew Weier O'Phinney wrote:

      
...
 Now, for an explanation of the "whys" behind this. As contributors,
 it's important to make atomic commits: your code fix should also
 include a corresponding unit test, and potentially a documentation
 commit as well. If we split the library and tests directories out
 into their own submodules, we cannot do so; the code and test must be
 committed separately, which is (a) more work, and (b) unintuitive.
 The same goes for related documentation commits.
        
-- Best Regards, Pieter Kokx PHP Developer Zend Framework developer
Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

weierophinney
Administrator
-- Pieter Kokx <[hidden email]> wrote
(on Wednesday, 31 March 2010, 05:43 PM +0200):

> Matthew Weier O'Phinney schreef:
>     -- Andy Fowler <[hidden email]> wrote
>     (on Monday, 22 March 2010, 11:38 AM -0400):
>         A quick note on "atomic commits" in git. One of the things I had to
>         learn when moving to Git was to get away from huge end-of-feature
>         commits, like one would make in SVN. Eventually, it becomes easier to
>         commit, revert, commit, revert many times during the course of a
>         feature. At the end of the feature branch you can squash those many
>         commits into a single atomic commit that contains all unit tests and
>         docs:
>
>
>     Yes, but that's exactly what I was getting at. If we do library/ and
>     tests/ and documentation/ all as submodules, we can't have that final
>     atomic commit that has the tests and docs and code all together. :)
>
>
> Well, actually you can. By making a commit to the super repository that
> includes the new commits to the submodules, you have that atomic commit that
> combines tests, code and docs.

Then you need to show me how to do it; I have yet to determine how this
would be achieved.

As an example, I did the following:

 * Created a repository for the library, which contains a single folder,
   Zend
 * Created a repository for the tests, which contains the unit test
   artifacts (phpunit.xml, Bootstrap.php, TestConfiguration.php.dist)
   and the Zend folder with all unit tests.
 * Created a "super" repository that uses git submodules to bring in the
   library and tests repositories.
 * Added a unit test, and code in the library to satisfy the unit test.

And this is where the problem occurs. To achieve the fix, you need to:

 * Commit the test
 * Commit the code
 * Update the submodules in the super repository
 * Commit the updated submodule definitions to the super repository

This is a hellacious process to go through for what is essentially a
single commit. I see it as a barrier to contribution.

If you can show that it can be done in a simpler fashion, I'm all ears.

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

till
In reply to this post by weierophinney
On Mon, Mar 22, 2010 at 1:36 PM, Matthew Weier O'Phinney
<[hidden email]> wrote:

> Greetings, all --
>
> Based on the wiki and the poll on this list, it looks like it makes
> sense to move forward with git. We're still trying to iron on some
> details about how this will work, but some details are falling into
> place.
>
> First, the things we plan:
>
>  * Git will be hosted on the ZF servers. This is so that we can have
>   full control over hooks (i.e., no network latency from web hooks),
>   integrate with other tools (e.g. viewgit, JIRA, etc.), and more. We
>   will offer both anonymous access as well as selective writable
>   access.
>
>   Some will say, "why not just use Github?" The above does not preclude
>   using github in any way; git allows you to specify multiple remotes
>   from your repository. I already plan to have a mirror of my personal
>   repo on github, as well as push to the ZF repo from the same
>   location. This will allow people to use some of githubs powerful
>   features such as pull queues while still allowing us to retain some
>   control over the repository. (It also insulates us from githubs
>   growing pains a bit -- I love github, but they are certainly
>   suffering from rapid growth the last few months!)
>
>  * We plan to have a read-only subversion mirror of the git repository.
>   This will allow those using svn:externals to continue to do so, as
>   well as allow folks more familiar with svn to continue to provide
>   patches.
>
> Next, things we're still researching:
>
>  * How to handle contributions. Since we require a signed CLA, we need
>   some way of verifying that a patch came from an author who has signed
>   the CLA. There have been some suggestions that this may be possible
>   with some sort of pre-commit verifications when committing to the
>   master repository (or when someone with commit privileges commits to
>   their own repository). We're still researching how this might be
>   implemented.
>
>  * Submodules...
>   For those unfamiliar with git submodules, they are roughly equivalent
>   to svn:externals. I say "roughly", because they differ in many ways:
>   they have to be manually initialized on each person's repository
>   (though they can be defined in the master repository); they need to
>   be manually updated periodically (unlike svn:externals, where anytime
>   you do an "svn up", it also syncs the external); etc.

http://github.com/blog/626-announcing-svn-support

I know Zend will do their own git, but since it's trivial to make a
mirror on Github, this might be something interesting for everyone who
misses svn:externals. ;-)

Till

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

Eric Clemmons
I totally thought it was an April Fool's joke, until it actually worked :)

Well, that's one thing solved with the Git migration.  Back to __invoke()! :)

On Apr 1, 2010, at 9:30 AM, till wrote:

> On Mon, Mar 22, 2010 at 1:36 PM, Matthew Weier O'Phinney
> <[hidden email]> wrote:
>> Greetings, all --
>>
>> Based on the wiki and the poll on this list, it looks like it makes
>> sense to move forward with git. We're still trying to iron on some
>> details about how this will work, but some details are falling into
>> place.
>>
>> First, the things we plan:
>>
>>  * Git will be hosted on the ZF servers. This is so that we can have
>>   full control over hooks (i.e., no network latency from web hooks),
>>   integrate with other tools (e.g. viewgit, JIRA, etc.), and more. We
>>   will offer both anonymous access as well as selective writable
>>   access.
>>
>>   Some will say, "why not just use Github?" The above does not preclude
>>   using github in any way; git allows you to specify multiple remotes
>>   from your repository. I already plan to have a mirror of my personal
>>   repo on github, as well as push to the ZF repo from the same
>>   location. This will allow people to use some of githubs powerful
>>   features such as pull queues while still allowing us to retain some
>>   control over the repository. (It also insulates us from githubs
>>   growing pains a bit -- I love github, but they are certainly
>>   suffering from rapid growth the last few months!)
>>
>>  * We plan to have a read-only subversion mirror of the git repository.
>>   This will allow those using svn:externals to continue to do so, as
>>   well as allow folks more familiar with svn to continue to provide
>>   patches.
>>
>> Next, things we're still researching:
>>
>>  * How to handle contributions. Since we require a signed CLA, we need
>>   some way of verifying that a patch came from an author who has signed
>>   the CLA. There have been some suggestions that this may be possible
>>   with some sort of pre-commit verifications when committing to the
>>   master repository (or when someone with commit privileges commits to
>>   their own repository). We're still researching how this might be
>>   implemented.
>>
>>  * Submodules...
>>   For those unfamiliar with git submodules, they are roughly equivalent
>>   to svn:externals. I say "roughly", because they differ in many ways:
>>   they have to be manually initialized on each person's repository
>>   (though they can be defined in the master repository); they need to
>>   be manually updated periodically (unlike svn:externals, where anytime
>>   you do an "svn up", it also syncs the external); etc.
>
> http://github.com/blog/626-announcing-svn-support
>
> I know Zend will do their own git, but since it's trivial to make a
> mirror on Github, this might be something interesting for everyone who
> misses svn:externals. ;-)
>
> Till

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

weierophinney
Administrator
In reply to this post by till
-- till <[hidden email]> wrote
(on Thursday, 01 April 2010, 04:30 PM +0200):

> On Mon, Mar 22, 2010 at 1:36 PM, Matthew Weier O'Phinney
> <[hidden email]> wrote:
> >  * Submodules...
> >   For those unfamiliar with git submodules, they are roughly equivalent
> >   to svn:externals. I say "roughly", because they differ in many ways:
> >   they have to be manually initialized on each person's repository
> >   (though they can be defined in the master repository); they need to
> >   be manually updated periodically (unlike svn:externals, where anytime
> >   you do an "svn up", it also syncs the external); etc.
>
> http://github.com/blog/626-announcing-svn-support
>
> I know Zend will do their own git, but since it's trivial to make a
> mirror on Github, this might be something interesting for everyone who
> misses svn:externals. ;-)

Interesting -- I thought this was an April Fools joke, but tried it,
and, indeed, it works. There's one problem, however: you can only
checkout the entire repo -- you can't checkout a subtree within the
repo. But it would make it trivial to maintain a read-only SVN mirror --
from which you *could* do a sparse checkout.

Thanks for the heads-up!

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Status: 22-Mar-2010 - Git organization

A.J. Brown-3
I recommend anyone who's not keen on git to read the following book:

http://progit.org/book/

I have the printed copy, and it's a fabulous read.


On Thu, Apr 1, 2010 at 10:58 AM, Matthew Weier O'Phinney
<[hidden email]> wrote:

> -- till <[hidden email]> wrote
> (on Thursday, 01 April 2010, 04:30 PM +0200):
>> On Mon, Mar 22, 2010 at 1:36 PM, Matthew Weier O'Phinney
>> <[hidden email]> wrote:
>> >  * Submodules...
>> >   For those unfamiliar with git submodules, they are roughly equivalent
>> >   to svn:externals. I say "roughly", because they differ in many ways:
>> >   they have to be manually initialized on each person's repository
>> >   (though they can be defined in the master repository); they need to
>> >   be manually updated periodically (unlike svn:externals, where anytime
>> >   you do an "svn up", it also syncs the external); etc.
>>
>> http://github.com/blog/626-announcing-svn-support
>>
>> I know Zend will do their own git, but since it's trivial to make a
>> mirror on Github, this might be something interesting for everyone who
>> misses svn:externals. ;-)
>
> Interesting -- I thought this was an April Fools joke, but tried it,
> and, indeed, it works. There's one problem, however: you can only
> checkout the entire repo -- you can't checkout a subtree within the
> repo. But it would make it trivial to maintain a read-only SVN mirror --
> from which you *could* do a sparse checkout.
>
> Thanks for the heads-up!
>
> --
> Matthew Weier O'Phinney
> Project Lead            | [hidden email]
> Zend Framework          | http://framework.zend.com/
> PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
>



--
A.J. Brown
Software Engineer, ZCE
blog : http://ajbrown.org
talk  : (937) 540-0099
chat : IntypicaAJ