Poll: direct to release, or Beta/RC phase for 2.2.0?

classic Classic list List threaded Threaded
14 messages Options
Reply | Threaded
Open this post in threaded view
|

Poll: direct to release, or Beta/RC phase for 2.2.0?

weierophinney
Administrator
For 2.1.0, I tried an experiment: I omitted the beta and RC phase from
the release process. My rationale was that in previous minor/major
releases, we've had quite limited engagement; no matter how many
betas/RCs we release, there's always a few bugs that are only found
after the stable release is announced. I translated this to, "Nobody
tests until they hear it's stable."

That said, we had a number of issues with the 2.1.0 release that
likely could have been avoided with even just a little more userland
testing.

Hence, the subject of this email:

* Skip the beta/RC phases entirely, and go straight to 2.2.0?
* Have a beta and/or RC phase prior to the 2.2.0 stable release?

Please reply on this thread, and, most importantly, indicate *why* you
vote the way you do. I'm truly curious.

(And yes: the 2.2.0 release train is still scheduled for the end of
the month. A handful of features are already in, several are in PRs
currently, and a few are in the wings. I'll be announcing code freeze
in 2-3 weeks.)

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

robertbasic
Have at least one beta release, and yes, not many of us will test the beta, and there is the possibility that people will find a lot more bugs once we hit stable, but! If we go directly to stable and hit bugs, people will complain that we didn't have a beta release. This way it's "we did best we could" and not "we're too lazy to test properly".

On the other hand, I don't think we have enough new features to warrant RCs (I might wrong with this one).

So, my vote is +1 for beta, reason is to avoid public bashing.


On 3 April 2013 19:54, Matthew Weier O'Phinney <[hidden email]> wrote:
For 2.1.0, I tried an experiment: I omitted the beta and RC phase from
the release process. My rationale was that in previous minor/major
releases, we've had quite limited engagement; no matter how many
betas/RCs we release, there's always a few bugs that are only found
after the stable release is announced. I translated this to, "Nobody
tests until they hear it's stable."

That said, we had a number of issues with the 2.1.0 release that
likely could have been avoided with even just a little more userland
testing.

Hence, the subject of this email:

* Skip the beta/RC phases entirely, and go straight to 2.2.0?
* Have a beta and/or RC phase prior to the 2.2.0 stable release?

Please reply on this thread, and, most importantly, indicate *why* you
vote the way you do. I'm truly curious.

(And yes: the 2.2.0 release train is still scheduled for the end of
the month. A handful of features are already in, several are in PRs
currently, and a few are in the wings. I'll be announcing code freeze
in 2-3 weeks.)

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc



--
~Robert Basic;
http://robertbasic.com/
Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

Sascha-Oliver Prolic
Same thoughts as Robert Basic,

as soon as we have feature freeze we should create a beta tag, test a week, fix bugs and then release.

+1 for beta first


2013/4/3 Robert Basic <[hidden email]>
Have at least one beta release, and yes, not many of us will test the beta, and there is the possibility that people will find a lot more bugs once we hit stable, but! If we go directly to stable and hit bugs, people will complain that we didn't have a beta release. This way it's "we did best we could" and not "we're too lazy to test properly".

On the other hand, I don't think we have enough new features to warrant RCs (I might wrong with this one).

So, my vote is +1 for beta, reason is to avoid public bashing.


On 3 April 2013 19:54, Matthew Weier O'Phinney <[hidden email]> wrote:
For 2.1.0, I tried an experiment: I omitted the beta and RC phase from
the release process. My rationale was that in previous minor/major
releases, we've had quite limited engagement; no matter how many
betas/RCs we release, there's always a few bugs that are only found
after the stable release is announced. I translated this to, "Nobody
tests until they hear it's stable."

That said, we had a number of issues with the 2.1.0 release that
likely could have been avoided with even just a little more userland
testing.

Hence, the subject of this email:

* Skip the beta/RC phases entirely, and go straight to 2.2.0?
* Have a beta and/or RC phase prior to the 2.2.0 stable release?

Please reply on this thread, and, most importantly, indicate *why* you
vote the way you do. I'm truly curious.

(And yes: the 2.2.0 release train is still scheduled for the end of
the month. A handful of features are already in, several are in PRs
currently, and a few are in the wings. I'll be announcing code freeze
in 2-3 weeks.)

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc



--
~Robert Basic;
http://robertbasic.com/



--
Sascha-Oliver Prolic
Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

Artur Bodera
In reply to this post by weierophinney

On Wed, Apr 3, 2013 at 7:54 PM, Matthew Weier O'Phinney <[hidden email]> wrote:

* Skip the beta/RC phases entirely, and go straight to 2.2.0?
* Have a beta and/or RC phase prior to the 2.2.0 stable release?

After a feature freeze, let's have a _fixed_ beta period of a few weeks (i.e. 3 weeks) during which only bug fixes are accepted and merged. This is the period where people can test out the beta in their apps (just call it a BETA, branch it or perpetually tag). This is also the period when contributors are encouraged not to focus their time on writing/reviewing new features, but testing the hell out of existing ones.

Before the 3 week deadline, there should be a visible drop of activity in PR queue (read: almost all bugs are remedied and little to none new bug-related PRs are flying in.). This means we have a "stable" release - a tag is created.


Cheers.

--
      __
     /.)\   +48 695 600 936
     \(./   [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

Mike Willbanks
In reply to this post by weierophinney
On Wed, Apr 3, 2013 at 10:54 AM, Matthew Weier O'Phinney <[hidden email]> wrote:
For 2.1.0, I tried an experiment: I omitted the beta and RC phase from
the release process. My rationale was that in previous minor/major
releases, we've had quite limited engagement; no matter how many
betas/RCs we release, there's always a few bugs that are only found
after the stable release is announced. I translated this to, "Nobody
tests until they hear it's stable."

That said, we had a number of issues with the 2.1.0 release that
likely could have been avoided with even just a little more userland
testing.

Hence, the subject of this email:

* Skip the beta/RC phases entirely, and go straight to 2.2.0?
* Have a beta and/or RC phase prior to the 2.2.0 stable release?


I don't believe that a "beta" will really help.  What people are describing of a feature freeze is really then an RC.  Betas still may receive features and as such IMO the "develop" branch is the beta.  I generally am always on the develop branch with my daily development.  I would like to see an RC phase in which we fix and resolve issues (maybe a week long).  

I do think we need documentation on how to test out an RC by using composer and that may provide additional devs to test it.  Issues that are found during the RC period would determine when the release would go out.

 

Please reply on this thread, and, most importantly, indicate *why* you
vote the way you do. I'm truly curious.

(And yes: the 2.2.0 release train is still scheduled for the end of
the month. A handful of features are already in, several are in PRs
currently, and a few are in the wings. I'll be announcing code freeze
in 2-3 weeks.)

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

robertbasic



On 3 April 2013 21:56, Mike Willbanks <[hidden email]> wrote:
I don't believe that a "beta" will really help.  What people are describing of a feature freeze is really then an RC.  Betas still may receive features and as such IMO the "develop" branch is the beta.  I generally am always on the develop branch with my daily development.  I would like to see an RC phase in which we fix and resolve issues (maybe a week long).  

Maybe not from the point of view of number of bugs found, but from a "marketing" point of view. You are an active contributor, and don't take this the wrong way, but you are not the target person for these releases. :) Take for example our Regular "Dev" Joe who wants to introduce ZF2 in his company. Maybe the management/team leads have a word or two on what tools are used during development. ZF2 might come down as irresponsible for just pushing out "releases full of bugs without proper testing period". That's what I'd like to avoid, even though we can't avoid people start testing when a stable comes out. That's out of our hands.

Also, the develop branch is at tops an alpha state in my opinion.
 

I do think we need documentation on how to test out an RC by using composer and that may provide additional devs to test it.  Issues that are found during the RC period would determine when the release would go out.

+1 on this!
Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

weierophinney
Administrator
In reply to this post by Artur Bodera
On Wed, Apr 3, 2013 at 2:51 PM, Artur Bodera <[hidden email]> wrote:

>
> On Wed, Apr 3, 2013 at 7:54 PM, Matthew Weier O'Phinney <[hidden email]>
> wrote:
>>
>>
>> * Skip the beta/RC phases entirely, and go straight to 2.2.0?
>> * Have a beta and/or RC phase prior to the 2.2.0 stable release?
>
>
> After a feature freeze, let's have a _fixed_ beta period of a few weeks
> (i.e. 3 weeks) during which only bug fixes are accepted and merged. This is
> the period where people can test out the beta in their apps (just call it a
> BETA, branch it or perpetually tag). This is also the period when
> contributors are encouraged not to focus their time on writing/reviewing new
> features, but testing the hell out of existing ones.
>
> Before the 3 week deadline, there should be a visible drop of activity in PR
> queue (read: almost all bugs are remedied and little to none new bug-related
> PRs are flying in.). This means we have a "stable" release - a tag is
> created.

I'd argue 3 weeks is much too long.

The develop branch is generally stable. All bugfixes applied to master
are also applied to develop, which means that new features are really
the only things not having userland testing. Since we require unit
tests for new features, and these are run for every push to master, we
have stable code; the main question is whether or not the new code
meets expectations of users -- which is why we need more testing.

We get around 40 - 70 NEW pull requests a week, even with stable
versions. They range from docblock typo fixes to full-fledged bugfixes
to new features. A prolonged beta and/or RC period will create an
immense backlog, as there will be many issues we choose to postpone.
If we don't postpone them, we have to release additional betas/RCs,
prolonging the cycle.

Considering we do monthly maintenance releases, I'd be happy with 1 to
2 weeks tops for any release cycle.

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

weierophinney
Administrator
In reply to this post by robertbasic
On Wed, Apr 3, 2013 at 3:06 PM, Robert Basic <[hidden email]> wrote:
> Also, the develop branch is at tops an alpha state in my opinion.

Why do you feel this?

As noted in a reply to Artur, all bugfixes for master are also merged
to develop, which means that the only "new" bits are new features.
These always have thorough unit test suites, which means that in terms
of initial requirements, they are quite stable.

What makes you feel they are alpha at best? (Truly curious, as I'm
wondering if I have a skewed view.)

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

weierophinney
Administrator
In reply to this post by Mike Willbanks
On Wed, Apr 3, 2013 at 2:56 PM, Mike Willbanks <[hidden email]> wrote:
> I do think we need documentation on how to test out an RC by using composer
> and that may provide additional devs to test it.  Issues that are found
> during the RC period would determine when the release would go out.

I have this planned; it's relatively easy to describe the process, and
will do this in any announcements we make.

I also agree with the assertion that we call them release candidates
vs. betas, particularly since we plan to start the release train after
we know the features are complete already.

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

cgmartin
In reply to this post by Mike Willbanks

On Wed, Apr 3, 2013 at 3:56 PM, Mike Willbanks <[hidden email]> wrote:
I would like to see an RC phase in which we fix and resolve issues (maybe a week long).

+1 for this. From a 3rd-party module author's point of view, it would be helpful to have a window to test and resolve any BC changes (if any), to be in sync with the release. Personal preference is 2 weeks, but a week would still beneficial.

Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

robertbasic
In reply to this post by weierophinney

On 3 April 2013 22:22, Matthew Weier O'Phinney <[hidden email]> wrote:
On Wed, Apr 3, 2013 at 3:06 PM, Robert Basic <[hidden email]> wrote:
> Also, the develop branch is at tops an alpha state in my opinion.

Why do you feel this?

I'm not really that active in the development process, so probably *my* view is skewed.

Well, in develop branch my default expectation is that something can be broken and that's how I approach it. If something is broken there then it's probably because it is still in development. With master/beta/RC I approach it as it's stable and if something is broken, I approach it as a bug.

Development branch is "Let's do some experimenting!"

Master/beta is "Let's see if this works!"

That's how I see it, feel it.

And again, these beta releases should not be directed towards active contributors, but to wake up us who are less active, have a note with things like "X and Y are new features, Z has been rewritten to better do it's job, W and Q have these bugs fixed.".

Plus, I truly believe having a week or so long beta period can look better from a marketing point of view. Oh, and I agree that 3 weeks is too long, a week, or a week and a half tops should be OK.
 

As noted in a reply to Artur, all bugfixes for master are also merged
to develop, which means that the only "new" bits are new features.
These always have thorough unit test suites, which means that in terms
of initial requirements, they are quite stable.

What makes you feel they are alpha at best? (Truly curious, as I'm
wondering if I have a skewed view.)

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc



--
~Robert Basic;
http://robertbasic.com/
Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

akrabat
In reply to this post by weierophinney

On 3 Apr 2013, at 18:54, Matthew Weier O'Phinney <[hidden email]> wrote:
>
> Hence, the subject of this email:
>
> * Skip the beta/RC phases entirely, and go straight to 2.2.0?
> * Have a beta and/or RC phase prior to the 2.2.0 stable release?


I'd like to see an RC phase for 1 week before release of 2.2.0. This gives a chance to find show-stoppers without stopping-up the processes for too long. I doubt anything longer than a week will make a difference as there's not that many people testing the non-released code.

Potentially we could even do the RC on master to make it easier for testers.

Regards,

Rob...
Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

Marcel Araujo
In reply to this post by weierophinney
+1


2013/4/3 Matthew Weier O'Phinney <[hidden email]>
For 2.1.0, I tried an experiment: I omitted the beta and RC phase from
the release process. My rationale was that in previous minor/major
releases, we've had quite limited engagement; no matter how many
betas/RCs we release, there's always a few bugs that are only found
after the stable release is announced. I translated this to, "Nobody
tests until they hear it's stable."

That said, we had a number of issues with the 2.1.0 release that
likely could have been avoided with even just a little more userland
testing.

Hence, the subject of this email:

* Skip the beta/RC phases entirely, and go straight to 2.2.0?
* Have a beta and/or RC phase prior to the 2.2.0 stable release?

Please reply on this thread, and, most importantly, indicate *why* you
vote the way you do. I'm truly curious.

(And yes: the 2.2.0 release train is still scheduled for the end of
the month. A handful of features are already in, several are in PRs
currently, and a few are in the wings. I'll be announcing code freeze
in 2-3 weeks.)

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc



--

Marcel Araujo
Analista de Sistemas
Desenvolvedor PHP/Zend/JavaScript/jQuery/NodeJS
Linux User #490101

http://www.twitter.com/marcelarauj0
http://blog.marcelaraujo.me
http://br.linkedin.com/in/marcelaraujo

Reply | Threaded
Open this post in threaded view
|

Re: Poll: direct to release, or Beta/RC phase for 2.2.0?

Marco Pivetta
In reply to this post by akrabat
More than a Beta phase, an RC phase encourages people to upgrade and thus can trigger a lot of problems that we didn't think of. There's a couple of pull requests that touch vital components of the MVC, and we _NEED_ to know if there's any unforeseen  consequence in their development. An example is the crashes with sessions when upgrading from 2.0 to 2.1. While the component worked fine, the upgrade invalidated existing sessions, which is something we couldn't know of.

I see some use for a beta phase for a major release, but I don't think people are not going to do the jump with BETA as much as with RC.

It's perfectly ok to freeze everything explicitly with a beta tag (makes it also easier to see if any BC break went in between beta tag and release by mistake), but the RC tag is needed (maybe one week later?) to spread eager adoption in my opinion.

If there's any module developers out there reading, please remember to use `"minimum-stability": "dev"` in your Travis-CI build configuration: that helps a lot!



On 3 April 2013 23:29, Rob Allen <[hidden email]> wrote:

On 3 Apr 2013, at 18:54, Matthew Weier O'Phinney <[hidden email]> wrote:
>
> Hence, the subject of this email:
>
> * Skip the beta/RC phases entirely, and go straight to 2.2.0?
> * Have a beta and/or RC phase prior to the 2.2.0 stable release?


I'd like to see an RC phase for 1 week before release of 2.2.0. This gives a chance to find show-stoppers without stopping-up the processes for too long. I doubt anything longer than a week will make a difference as there's not that many people testing the non-released code.

Potentially we could even do the RC on master to make it easier for testers.

Regards,

Rob...