Modular architecture != architecture for modules

classic Classic list List threaded Threaded
11 messages Options
Reply | Threaded
Open this post in threaded view
|

Modular architecture != architecture for modules

Artur Bodera
Hello again.

Because I've been pointed at (along with Kevin) during last IRC meeting, I'd
like
to continue a very important IMO discussion on one topic.

I've spent some time with Kevin on IRC later that night because I wanted to
get a better glimpse on what was his perception on problems and scale of
problems that, in his opinion, ZF project has.

Today, I'll try to bring up the first challenge up to discussion.

I'm also personally unsatisfied by how the "module vs component" discussion
on irc has been muted, and how Matthew flattened it by pointing at us as the
"only 2 people that had problem with the definition". Fortunately here, I
have
a lot more space to explain in more detail, what many of you are missing out
in this discussion and there is no one to "vote me out" ;-P

The discussion was NOT about definitions. Neither I or Kevin are stupid, we
clearly understand the difference between "collection of code that solves a
more specific" and "collection of code that solves a more specific atomic.."
(besides those being very flexible in interpretation).

The most important message I'd like to bring up today is the difference
between a "modular architecture" of a project and an "architecture for
modules".

[ Those who just started picking their noses, "meh"-ing and calling me
Mr Obvious - you are the most important recipients of this message.  ]




Modules: I personally use mvc modules in ZF1 since the dawn of
Zend_Application. Even though they didn't get much lovin and are a
mess (i.e. boostrap re-loads/double-loads the same resources). There
has been some evolution in the thinking, as now "modules"  can have
assets, configuration, routes, etc. and these will be nicely, automatically
handled, merged and resolved by MVC+DI.

This is what you call "and architecture for modules". ZF2 will have more
tools than ZF1 to handle installation of blobs of code, images and other
stuff. It will be able to do tasks I've mentioned. ZF2 will be a foundation
to build MVC modules on.

Kevin named ZF2:  "ZF1 with modules"

It is more hurtful than you might think.


What is a modular architecture then?

Notice how I previously wrote "modular architecture for a PROJECT". What I
mean by that, is that modular philosophy crosses and shoots through all
levels of A PROJECT. What does that mean in practice?

 * flexibility
    Everything is optional. MVC is optional, Ralph's DB is optional, Form
    is optional, Tool, Autoloading and Application is optional (I know what
    I'm poking :) ).

 * extendability and decoupling
    There is no single component, or there are very few that cannot be
    extended and/or replaced with a custom implementation. All
    components are equal in rights, ZF-reviewed or not.  All are
    obtainable and distributed the same way.

 * consistency
    Because of obvious marketing and educational reasons, there should
    be a "popular", "standard" distribution of a framework, but at
    architectural level (with everything optional) all pieces of the puzzle
are
    replaceable. ZF2 standard distribution becomes an example of how
    thin the ZF2 core is and how any modules/components can be bundled
    even if they don't have to be.

 * openness
    Openness (h@ck4bility) is a door to innovation. A community project with
    modular architecture invites people to write their stuff. Most of it
will be
    crappy in quality, many will be worthwhile a few will shine.


There is an obvious conflict between code quality standards, legal issues
(CLA)
and that openness. I've had bricks thrown at my by Paddy several times when
mentioning of having 3rd party components in the Zend\* namespace, as if the
PHP 5.3 NS now became the international catalog of brand's code quality
ranking something.

I am a developer who values good unit tests, stability and license
transparency.
On the other hand, I cannot overlook what recent years have shown us with
other
agile projects. Dev. community has noticed, that they have a predictable
probability
distribution in terms of code quality. Although I don't have solid data for
that, from
my experience projects like drupal, jquery or even pear, comply to the
90-9-1 theory.
(http://www.wikipatterns.com/display/wikipatterns/90-9-1+Theory)

This means that after we create (which has been "voted over" :P) a
marketplace,
catalog for 3rd party components, 90% will potentially be crap, 9% will be
very
popular, high-quality functional components that will shape the perceived
value of
the ecosystem, and 1% will be "gems" - hero components that will be a
presented on keynotes and will build the brand for the project.


I'm also a man of action. I'm not a fan of ranting just to point out what's
broken.
I prefer to have a concept, solution proposition before I step out. Here is
what I
propose:

1. THIN CORE
In order to meet the need of flexibility, we need a stable but very thin
core. This
means a collection of "components" that are foundation for other components.
Core can also maintain and handle any number of additional
modules/components.

2. CONSISTENCY AND SIMPLIFICATION
Review the concept of "module == component". This will be particularly
painful
for all purists, but please try to look through it. It's (again) not about
naming
conventions but about __architecture__
A "module == component" is about having a common name for the concept
of reusable code. Here is an example

 * MVC module
 * Security module
 * Zend\Db\Adapter\Firebird module
 * Zend\Form\Element\Video module (or VideoElement module)

The goal of this approach is simplify.
instead of:
    "framework consisting of components, assets, meta-modules, packages,
adapters, ...."
think about:
    "framework consisting of number of modules",

"Everything except for core is a module" will also meet the requirement
of openness and consistency. Because both zf-contributors modules
and "3rd party" modules are on thesame page, it is a truly modular
architecture. Before bashing this argument, please take a moment to browse
through Drupal and Wordpress. Because these are CMSes, their core is not
thin by
ZF-standards, but it is ridiculously thin by CMS standards. And, everything
in these
projects is a drop-in module (also, peek again at the sheer number of
modules
available for download for these projects)

Consistent naming convention will greatly help with education.
Instead of saying:
    - grab the FormElements module, then download Captcha component
      and then look for ReCaptcha adapter which goes to xyz folder"

we'll just say:
    - install ReCaptcha module


3. THERE IS NO 3RD PARTY.  (also "Everything is miscellaneous").
To meet the requirement of openness, there can be no wall between us (????),
contributors and other people who just had an account set up on github, or
email at gmail. Contributing code is not the same as writing stupid
comments
on facebook or uploading cat photos on icanhazcheeseburger. It takes more
knowledge and time, so there will be less "spam". In terms of code quality,
this is what GUIDELINES are for.  We can have both! Openness and code
quality guidelines.
We _must not_ have a distinction between "us" and "them". There is no
"them".
We _must not_ have a wall of review process. The market does that for us.
People will do that for us.

I'm sorry, but trying to say that each and every ZF1 class is an example of
engineering brilliance is a huge overstatement ;) How does it translate to
real
world? Well, some people just don't use Zend_Db or Zend_Form, or Zend_Mail.
Same goes with drupal. Many "standard modules" have been replaced by higher
quality modules from the catalog, from "the community". Now they are part
of
core or are bundled in standard distribution.

Modular architecture in this regard means, that each we give people a choice
of which and whose module I'll use for forms, and db and mailing people.

What about current zend-review components? Put them inside
repository/catalog
ALONG with all other components. Because the CR team has put a lot of
effort and their time, we can have a [zend-certified] badge or some other
form
of letting people know that these components are _different_.
NOT "better"... "different".  You can say "featured" for a lack of better
word at
this time.

I'm sorry, but assuming that ZF review process somewhat makes ZF
components better than any external person in the world would write them is
just mathematically nearly impossible :-) I'm against discrimination which
will discourage any contributors that might just find a better way to solve
XYZ problem that the "almighty zf-contributor ML subscribers".



If ZF2 became an open, modular, component framework it will create an
ecosystem. Worst mistake of Steve Jobs with iPhone 1 was that the API was
not available since day 1. Opening ZF2 doors to innovation will allow it to
grow
and will hopefully invite back all those who were once smiling to us from
http://framework.zend.com/community/contributors. Of course this isn't all
for them, but mostly for current and future users of the framework.



IMO the biggest challenge is not the architecture. We are smart and will
figure
it out. The biggest challenge is breaking conceptual and marketing BC :-)
We, as engineers, understand that some things need to be destroyed to make
space for new ones. I hope my message will induce a discussion around that
topic.



--
      __
     /.)\   +48 695 600 936
     \(./   [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Modular architecture != architecture for modules

akrabat
Hi,

There's a lot here and I'm not sure that I actually understand it all. I'm gonna take a stab at responding though.

On 1 Sep 2011, at 10:59, Artur Bodera wrote:

> I'm also personally unsatisfied by how the "module vs component" discussion
> on irc has been muted, and how Matthew flattened it by pointing at us as the
> "only 2 people that had problem with the definition". Fortunately here, I
> have
> a lot more space to explain in more detail, what many of you are missing out
> in this discussion and there is no one to "vote me out" ;-P
>
> The discussion was NOT about definitions. Neither I or Kevin are stupid, we
> clearly understand the difference between "collection of code that solves a
> more specific" and "collection of code that solves a more specific atomic.."
> (besides those being very flexible in interpretation).

It's really really important to always bear in mind that in Zend Framework, the word "component" has a specific meaning which cannot be changed without significant confusion. i.e. a component is the name of the top level of groups of related classes within a library. i.e. Zend\Db is a component. Zend\Db\Adapter\Pdo\Mysql is not.

We can't get away from our history and magically rewrite so that we can use the word component in any other manner.

Of course, modules is also in use within ZF, but much less ingrained. I personally preferred Paddy's "zundles" tongue-in-cheek word to describe what we're trying to do with ZF2 "groupings of related files". However, I think most people appear to be happy with us (re)using the word "module" for this.  I think this is because in practice, most ZF2 modules created will interact with the MVC system to some degree or another.

To put it another way, never use the word component when talking about the ZF2 application architecture and we're all much more likely to stay on the same page.


> The most important message I'd like to bring up today is the difference
> between a "modular architecture" of a project and an "architecture for
> modules".

I'm not sure there's that much difference. In either case you need some sort of module management/bootstrapping/initialisation/call it what you want system to get it all going. If the MVC is that system, then it also brings a lot more capability to the table.


Arguably, though, the module initialisation/configuration should be separate from MVC and I've raised the idea of Zend\Module\Manager to handle this responsibility.


Another way of putting it is that I think that the concept of a installing and running  a module belongs in the Modules RFC, but the bits related to  zend\application, routes, dispatching, view overloading, layout handling etc belong in the "ZF2 MVC" RFC.
 


> What does that mean in practice?
>
> * flexibility
>    Everything is optional. MVC is optional, Ralph's DB is optional, Form
>    is optional, Tool, Autoloading and Application is optional (I know what
>    I'm poking :) ).

We have this with ZF1 and will have it with ZF2. Especially if module installation and config management is separate from Zend\Application. I don't know about Tool though as it depends on if we mandate cli for installation of modules.

> * extendability and decoupling
>    There is no single component, or there are very few that cannot be
>    extended and/or replaced with a custom implementation. All
>    components are equal in rights, ZF-reviewed or not.

This isn't application architecture. However, it has always been a ZF2 goal which is why we're removing Singletons and implementing DI.

>  All are obtainable and distributed the same way.

I would have thought that  the module management stuff  needs to belong in a library that isn't itself a module. This implies that you need to install ZF2 before you can install any modules. Especially if ZF2 supplies the mechanisms for installing a module.

I agree with the basic principle of pointing your install tool somewhere and you can pick up either an "official" ZF2 module or an 3rd party one. Ideally, the install tool will have something like PEAR's concept of channels.  I think this has been generally agreed within the community too.


> * consistency
>    Because of obvious marketing and educational reasons, there should
>    be a "popular", "standard" distribution of a framework,

Yes. A developer should have all that he needs "out of the box" in order to write a ZF2 application once he's installed ZF2.

> but at architectural level (with everything optional) all pieces of the puzzle
> are replaceable.

This is a repeat of what you said earlier in extendability and decoupling, I think?

> ZF2 standard distribution becomes an example of how
>    thin the ZF2 core is and how any modules/components can be bundled
>    even if they don't have to be.

I think what you are getting at here is that within the "What will the ZF2 distribution include?" RFC document, we should be considering that all components that are not in the core category should be implemented within Modules as defined by the Modules RFC?

What would the directory structure look like then? e.g. do I have to re-download the DB module every time I create a ZF2 application?

> * openness
>    Openness (h@ck4bility) is a door to innovation. A community project with
>    modular architecture invites people to write their stuff. Most of it
> will be
>    crappy in quality, many will be worthwhile a few will shine.
>
[snip]

>
> This means that after we create (which has been "voted over" :P) a
> marketplace,
> catalog for 3rd party components, 90% will potentially be crap, 9% will be
> very
> popular, high-quality functional components that will shape the perceived
> value of
> the ecosystem, and 1% will be "gems" - hero components that will be a
> presented on keynotes and will build the brand for the project.
>

If I have understood you correctly, this is already intended. i.e. We have agreed as a community that ZF2 will allow anyone to build and distribute modules and that the module installation tool will allow this to be as seamless as we can.  The concept of a "website containing a registered list of all available 3rd party modules" is welcome, but Zend will not provide it and it's up to someone in the community to build it and get traction for it.


> I'm also a man of action. I'm not a fan of ranting just to point out what's
> broken.
> I prefer to have a concept, solution proposition before I step out. Here is
> what I
> propose:
>
> 1. THIN CORE
> In order to meet the need of flexibility, we need a stable but very thin
> core. This
> means a collection of "components" that are foundation for other components.
> Core can also maintain and handle any number of additional
> modules/components.

Could you write down how you would organise all the ZF2 components as listed with the RFC in such as way as which ones are in your "thin core" and which ones are in modules and what those modules would be called?


> 2. CONSISTENCY AND SIMPLIFICATION
> Review the concept of "module == component". This will be particularly
> painful for all purists, but please try to look through it.

As noted above, the use of word "component" is done. However, what you actually are talking about here is this bit:

> "Everything except for core is a module"

I would phrase it as "Everything except for core is within a module", which I think may be worth exploring.

For the ZF2 distributed modules, each module wouldn't be doing anything except acting as a repository for its components. Oh, it would also define the dependencies on other modules which contain the dependent components of components in this module (if you see what I mean!).


>  Before bashing this argument, please take a moment to browse
> through Drupal and Wordpress. Because these are CMSes, their core is not
> thin by ZF-standards, but it is ridiculously thin by CMS standards. And, everything
> in these projects is a drop-in module (also, peek again at the sheer number of
> modules available for download for these projects)

Ironically, I would argues that WordPress is a great example of the opposite. With WP, plugins and themes are quite clearly subservient to the main application.  It's unimportant to the discussion of whether ZF2 should groups its components into modules though.

> Consistent naming convention will greatly help with education.
> Instead of saying:
>    - grab the FormElements module, then download Captcha component
>      and then look for ReCaptcha adapter which goes to xyz folder"
>
> we'll just say:
>    - install ReCaptcha module

This will be true anyway (see "openness" above). The ReCaptcha module would contain the relevant components to be inserted into the Form component. Of course, that assumes that ReCaptcha is distributed as a separate ZF2 module. It would certainly be true is ReCaptcha was a third-party module.


> 3. THERE IS NO 3RD PARTY.  (also "Everything is miscellaneous").
> To meet the requirement of openness, there can be no wall between us (????),
>
[snip]
> I'm sorry, but assuming that ZF review process somewhat makes ZF
> components better than any external person in the world would write them is
> just mathematically nearly impossible :-) I'm against discrimination which
> will discourage any contributors that might just find a better way to solve
> XYZ problem that the "almighty zf-contributor ML subscribers".

An official Zend Framework component is not merely related to the quality of the code. It is also related to support of that code by Zend Technologies and the community in liaison with the CR team. Also it has a CLA attached to it.  This is important to some users of ZF and will not change.

Having said that, the whole point of the module RFC installation section is to ensure we have a system where installing a 3rd party module is as seamless as possible regardless of where it has come from. This will result in what you want anyway.  e.g. if Ryan creates a Twig module then the commands to install it would be something like:

prompt>zf register-channel module zf2.rmauger.co.uk
prompt>zf install module rmauger/Twig

(or download from his site and copy into your modules/ folder!)

If Ryan's module is awesome then everyone uses that and ignores the default view renderer.



> I hope my message will induce a discussion around that
> topic.


In summary, I don't think that we're going to be very far from what you are wanting.


Regards,

Rob…



--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Modular architecture != architecture for modules

Xerkus

I have strong feeling slightly different concepts are mixed here.

I think we should split it into following:
1) Application modules(group of related models, services, resources,
configuration. Controllers, views as well). Or what is now being
defined by modules RFC.
2) ??? zundles! (And this part is actually required good name. Hi Kevin
:) )

Application modules can be part or specific type of "zundle" while
"zundle" can be module, set of javascripts or images, additional 3-d
party components, may be zf components too.

ps: may be i get it wrong. Haven't time yet to dig into prototypes and
stuff.

--

Aleksey (aka ~Xerkus)
PGP key | http://xerkus.ru/xerkus.asc

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Modular architecture != architecture for modules

weierophinney
Administrator
In reply to this post by Artur Bodera
-- Artur Bodera <[hidden email]> wrote
(on Thursday, 01 September 2011, 11:59 AM +0200):

> Because I've been pointed at (along with Kevin) during last IRC
> meeting, I'd like to continue a very important IMO discussion on one
> topic.
>
> I've spent some time with Kevin on IRC later that night because I
> wanted to get a better glimpse on what was his perception on problems
> and scale of problems that, in his opinion, ZF project has.
>
> Today, I'll try to bring up the first challenge up to discussion.
>
> I'm also personally unsatisfied by how the "module vs component"
> discussion on irc has been muted, and how Matthew flattened it by
> pointing at us as the "only 2 people that had problem with the
> definition". Fortunately here, I have a lot more space to explain in
> more detail, what many of you are missing out in this discussion and
> there is no one to "vote me out" ;-P

I'm sorry you feel you were muted. The point I was trying to make during
the meeting is that the meeting is for fleshing out final details and
making decisions. New ideas should be proposed via RFCs and dedicated ML
threads.

Several of us did exactly that in the last few weeks and threw out
proposals for how we define these concepts, and how we might move
forward. While you and Kevin both participated in these discussions,
neither of you put forth an alternate proposal. As such, all any of us
on the ML have is a piece-meal view of what either of you envision --
which makes discussion during the meetings very, very difficult. We have
nothing concrete to look at.

In the mean time, there _were_ concrete RFCs posted that we _could_
discuss and vote on.

Now, to continue: thank you for this post, as it finally puts forth a
more concrete vision to discuss. More below.

<snip>

> The most important message I'd like to bring up today is the
> difference between a "modular architecture" of a project and an
> "architecture for modules".

<snip>

> Modules: I personally use mvc modules in ZF1 since the dawn of
> Zend_Application. Even though they didn't get much lovin and are a
> mess (i.e. boostrap re-loads/double-loads the same resources). There
> has been some evolution in the thinking, as now "modules"  can have
> assets, configuration, routes, etc. and these will be nicely,
> automatically handled, merged and resolved by MVC+DI.
>
> This is what you call "and architecture for modules". ZF2 will have
> more tools than ZF1 to handle installation of blobs of code, images
> and other stuff. It will be able to do tasks I've mentioned. ZF2 will
> be a foundation to build MVC modules on.
>
> Kevin named ZF2:  "ZF1 with modules"
>
> It is more hurtful than you might think.
>
>
> What is a modular architecture then?
>
> Notice how I previously wrote "modular architecture for a PROJECT".
> What I mean by that, is that modular philosophy crosses and shoots
> through all levels of A PROJECT. What does that mean in practice?
>
>  * flexibility
>     Everything is optional. MVC is optional, Ralph's DB is optional,
>     Form is optional, Tool, Autoloading and Application is optional (I
>     know what I'm poking :) ).

This is already what we're planning, though. If you want to use a single
component, you grab it via the installer (whatever we end up choosing).
Want just Zend\Db?
   
    prompt> <installer> install zf\Zend_Db

Does this differ from what you mean/intend? If so, how?

>  * extendability and decoupling
>     There is no single component, or there are very few that cannot be
>     extended and/or replaced with a custom implementation. All
>     components are equal in rights, ZF-reviewed or not.  All are
>     obtainable and distributed the same way.

Again, this is also a plan. One guiding principle for ZF2 has been that
we try and follow SOLID principles when possible, and define interfaces
for anything we think could provide an extension point -- which is a lot
more often than not. Does this differ from what you mean/intend? If so,
how?

Regarding all components being equal, "ZF-reviewed or not," I don't know
that we can do that while we require a CLA, nor from a maintenance
perspective. One big point of the proposal process is to identify
components that we think will be of enough significant general use that
we can commit to maintaining them long-term. Simply put, if there's not
enough interest to maintain a component, we shouldn't be shipping it.

That said, I _do_ think components should be obtainable and distributed
in a consistent manner, whether or not they are an official part of the
ZF distribution, or part of the larger ZF user/developer ecosystem.
That's where finding a sane and powerful installation tool comes in;
any tool we choose should allow for registering channels or repository
sources in a central location to allow developers to search for
components to use. Does this differ from what you intend? If so, how?

>  * consistency
>     Because of obvious marketing and educational reasons, there should
>     be a "popular", "standard" distribution of a framework, but at
>     architectural level (with everything optional) all pieces of the
>     puzzle are replaceable. ZF2 standard distribution becomes an
>     example of how thin the ZF2 core is and how any modules/components
>     can be bundled even if they don't have to be.

This was discussed in-depth yesterday during the distribution portion of
the meeting, and I've updated the distribution RFC based on that
discussion. I personally feel that:

 * Providing an installable package per component, plus
 * A variety of meta-packages describing groupings of components

answers this well. Based on the IRC discussion yesterday, you seemed to
agree. Does this differ from what you intend? If so, how?

>  * openness
>     Openness (h@ck4bility) is a door to innovation. A community
>     project with modular architecture invites people to write their
>     stuff. Most of it will be crappy in quality, many will be
>     worthwhile a few will shine.
>
> There is an obvious conflict between code quality standards, legal
> issues (CLA) and that openness. I've had bricks thrown at my by Paddy
> several times when mentioning of having 3rd party components in the
> Zend\* namespace, as if the PHP 5.3 NS now became the international
> catalog of brand's code quality ranking something.

It's not just about quality; it's about what is considered code that the
ZF project _MAINTAINS_ and _SUPPORTS_. Having 3rd-party components share
the same namespace makes it difficult for end-users to know who to go to
for help, to report issues, or to submit patches.

Openness is fine, and I fully support having a community hub for modules
-- but we simply can't have folks using the "Zend" namespace for code
we're not providing directly from the ZF project repository, for the
reasons outlined in the previous paragraph.

The idea of a forge site for 3rd-party code has been raised a ton, and a
thread or three have even been started on this ML amongst folks
interested in building such a site. I'd love to see one; I also know
that my team has too-limited resources to do it while also working on
ZF2 and helping maintain ZF1. It has to be a community initiative at
this point; however, once something workable is in place, we can
certainly help support it in terms of providing a home (server and DNS)
for it, as well as access for maintainers to assist.

> I am a developer who values good unit tests, stability and license
> transparency.  On the other hand, I cannot overlook what recent years
> have shown us with other agile projects. Dev. community has noticed,
> that they have a predictable probability distribution in terms of code
> quality. Although I don't have solid data for that, from my experience
> projects like drupal, jquery or even pear, comply to the 90-9-1
> theory.
> (http://www.wikipatterns.com/display/wikipatterns/90-9-1+Theory)
>
> This means that after we create (which has been "voted over" :P) a
> marketplace, catalog for 3rd party components, 90% will potentially be
> crap, 9% will be very popular, high-quality functional components that
> will shape the perceived value of the ecosystem, and 1% will be "gems"
> - hero components that will be a presented on keynotes and will build
> the brand for the project.

And I'd argue that the "gems" and some of the "popular, high-quality"
components become candidates for inclusion in the main project itself.

I don't see any conflict here, to be honest. With DVCS, it becomes
trivial to merge such components into the master repository over time --
while allowing the projects to maintain independence if desired, but
also have independence while they work out a variety of kinks on a more
agile basis before they suggest inclusion in the main project.

> I'm also a man of action. I'm not a fan of ranting just to point out
> what's broken.  I prefer to have a concept, solution proposition
> before I step out. Here is what I propose:
>
> 1. THIN CORE
> In order to meet the need of flexibility, we need a stable but very
> thin core. This means a collection of "components" that are foundation
> for other components.  Core can also maintain and handle any number of
> additional modules/components.

This is planned. Do you disagree with the core as defined in the
distribution RFC? If so, what changes would you make?

> 2. CONSISTENCY AND SIMPLIFICATION
> Review the concept of "module == component". This will be particularly
> painful for all purists, but please try to look through it. It's
> (again) not about naming conventions but about __architecture__ A
> "module == component" is about having a common name for the concept of
> reusable code. Here is an example
>
>  * MVC module
>  * Security module
>  * Zend\Db\Adapter\Firebird module
>  * Zend\Form\Element\Video module (or VideoElement module)
>
> The goal of this approach is simplify.
> instead of:
>     "framework consisting of components, assets, meta-modules, packages,
> adapters, ...."
> think about:
>     "framework consisting of number of modules",

I actually find the above _more_ confusing, to be honest. Why wouldn't
all form-related classes be in a single module? and DB adapters? I could
understand if they are _third-party_ offerings, but within the framework
itself?

Also, you've misquoted. The framework does not consist of "components,
assets, meta-modules, packages, adapters, ...." That has never been
stated. The framework as _currently_ proposed, consists of components --
code, tests, and documentation only.

_Distribution_ would allow selecting individual components and/or
meta-packages consisting of multiple components. That's what the
Distribution RFC was trying to identify.

I think part of the problem I'm having is that I'm not seeing any
concrete details on what a module, as you conceive it, _looks_ like, in
the repository, nor how this impacts maintenance and packaging.

In the current paradigm, running tests is easy -- a single directory
contains tests for the entire framework, making it easy to offer a
single autoloader for loading all library and test assets during
testing. Because of the flexibility of PHPUnit, we can easily test a
single component or class, or use annotations to group components and/or
tests for running at once.

Building the manual is also similarly easy: a single manual.xml file is
used, and pulls in all manual pages via xincludes. Translation is fairly
easy to accomplish, as we can copy the english sources into each
translation directory, making it easy to run statistics on translation
coverage. Fallbacks can be defined for xincludes to allow falling back
to english if a translation is missing.

If I understand correctly, you're positing that each component becomes a
self-contained module, each containing code, tests, and documentation:

    Zend-Form/
        library/
            Zend/
                Form/
                    ....
        tests/
            Zend/
                Form/
                    ...
        documentation/
            manual.xml
            specs/
                ...

This has huge ramifications in terms of how the repository is
structured, how tests are run, and how the manual is built.

On the plus side, the entire component is in one place, and it's easy to
visualize what a package looks like for a discrete component. It would
play very easily into any packaging system and distribution tool we
might select.

On the flip side:

 * How does the component know where dependencies exist?
    * This is important both when installed, as well as when *running*
      *the* *tests*.
 * How does documentation work?
    * On the website, we'd want a single tree of manuals in order to
      promote easy discovery. This implies a single build system -- how
      would we accomplish that?
    * Within a single module, do we build an individual manual for that
      module for distribution? What about linking to other component
      documentation -- how are those links resolved, particularly if
      that component is not installed?

Another point that comes to mind: you show individual adapters as being
potentially modules -- why? Wouldn't that make things _more_ confusing?
Shouldn't grabbing a component also grab all subcomponents such as
adapters -- and why wouldn't we want these as a single package?

I can _definitely_ see the pros to such an architecture -- don't get me
wrong. I just also see a ton of HUGE infrastructure issues that would
need to be addressed, and I think we have a lot of work to iron those
out if we want to move forward. Frankly, I'm not sure anybody wants new
development lingering for weeks or months before we figure those out,
when the current infrastructure already is tried, tested, and proven.

Feel free to try and convince me otherwise, though -- as I said, I can
see the benefits; I'm just not sure they outweight the costs of change
currently.

> "Everything except for core is a module" will also meet the
> requirement of openness and consistency. Because both zf-contributors
> modules and "3rd party" modules are on thesame page, it is a truly
> modular architecture. Before bashing this argument, please take a
> moment to browse through Drupal and Wordpress. Because these are
> CMSes, their core is not thin by ZF-standards, but it is ridiculously
> thin by CMS standards. And, everything in these projects is a drop-in
> module (also, peek again at the sheer number of modules available for
> download for these projects)
>
> Consistent naming convention will greatly help with education.
> Instead of saying:
>     - grab the FormElements module, then download Captcha component
>       and then look for ReCaptcha adapter which goes to xyz folder"
>
> we'll just say:
>     - install ReCaptcha module

Um, I think you've convoluted the story significantly, and not really
detailed what you think you're trying to install here.

If you're wanting to install the ReCaptcha form element, I don't see why
it would be different than this:

    prompt> <installer> install zf\Zend_Form
    prompt> <installer> install zf\Zend_Service_ReCaptcha

I think this would be true even in the modular architecture you envision
-- the recaptcha form element consumes the recaptcha service as an
optional dependency (if missing, it obviously fails all over the place,
but you can still install the form component without it); the recaptcha
service can be used without the form component.

Can you provide a different example that better illustrates the
differences in what has been proposed vs what you envision, please?

> 3. THERE IS NO 3RD PARTY.  (also "Everything is miscellaneous").
> To meet the requirement of openness, there can be no wall between us
> (????), contributors and other people who just had an account set up
> on github, or email at gmail. Contributing code is not the same as
> writing stupid comments on facebook or uploading cat photos on
> icanhazcheeseburger. It takes more knowledge and time, so there will
> be less "spam". In terms of code quality, this is what GUIDELINES are
> for.  We can have both! Openness and code quality guidelines.  We
> _must not_ have a distinction between "us" and "them". There is no
> "them".  We _must not_ have a wall of review process. The market does
> that for us.  People will do that for us.

I've outlined above why I think we need to make a differentiation
between the official ZF distribution (the framework upon which
applications are built) and a community forge/hub/etc. I'm not going
over that again here.

> I'm sorry, but trying to say that each and every ZF1 class is an
> example of engineering brilliance is a huge overstatement ;)

ZF1 has evolved tremendously over time. Many of the testing and
documentation requirements also evolved -- most of these were missing
prior to the 1.0 release, meaning that older code often is not
engineered to the same standards that recent contributions are.

Additionally, the PHP landscape has changed a ton in the last 5.5 years
since the initial 0.1.0 release -- understanding and application of
patterns, testing practices, etc. have taken a quantum leap forward
since then. Yes, some code in ZF1 is better than others. The point is
that we have standards that we've come to expect that we can now apply
to the code base as a whole for ZF2.

> How does it translate to real world? Well, some people just don't use
> Zend_Db or Zend_Form, or Zend_Mail.  Same goes with drupal. Many
> "standard modules" have been replaced by higher quality modules from
> the catalog, from "the community". Now they are part of core or are
> bundled in standard distribution.

Having use-at-will installation answers your first question. Regarding
the second part of your paragraph, that's what having a new major
version allows us to do -- get rid of cruft and deprecated
functionality, and add new code.

> Modular architecture in this regard means, that each we give people a
> choice of which and whose module I'll use for forms, and db and
> mailing people.

I see no conflict here. Assuming an installer tool that speaks to a
server allowing developers to register their own repositories,
developers can search for and install any component they want, be it
from the official Zend repo or otherwise.

> What about current zend-review components? Put them inside
> repository/catalog ALONG with all other components. Because the CR
> team has put a lot of effort and their time, we can have a
> [zend-certified] badge or some other form of letting people know that
> these components are _different_.  NOT "better"... "different".  You
> can say "featured" for a lack of better word at this time.
>
> I'm sorry, but assuming that ZF review process somewhat makes ZF
> components better than any external person in the world would write
> them is just mathematically nearly impossible :-) I'm against
> discrimination which will discourage any contributors that might just
> find a better way to solve XYZ problem that the "almighty
> zf-contributor ML subscribers".

That's not the argument for inclusion in the ZF distribution AT ALL! The
argument for inclusion is CAN WE and WILL WE SUPPORT and MAINTAIN the
component long-term? I.e., is the component something that has a
demonstrated need, and for which we will be able to find new maintainers
should the original leave in the future?

An official repo can easily live alongside a community repo -- or
distributed individual repos. We can setup infrastructure that allows
folks to search for functionality they desire from their installer, and
which searches all registered repos. In that situation, the ZF repo is
simply _one_ _source_ -- one that many modules/components/whatever might
list dependencies against, but simply one source. This allows somebody
to distribute their own form component, or DB implementation, or ORM, or
whatever, and have it installed directly in their app or alongside the
ZF installation (or set of ZF components they have installed).

> If ZF2 became an open, modular, component framework it will create an
> ecosystem.

Yes, we need an ecosystem. I argue the way to do that is to be smart
about selecting a packaging and installation tool that allows us to
aggregate repos from disparate sources, including the ZF repo.

The current processes are in place to create a story for maintenance and
longevity of the code shipped directly with the framework. How
individuals version and maintain their own repositories is up to them;
we can simply make it easy for them to get their code in the hands of
users.

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Modular architecture != architecture for modules

Kevin McArthur-2
Matthew,

I'll leave the rest of the discussion to you folks and the community,
but I have to respond to one specific statement:

While you and Kevin both participated in these discussions,
neither of you put forth an alternate proposal.

That's absolutely not true. I've penned dozens (potentially hundreds) of
emails to this list on the concepts I envision. I've contributed
significantly to the ZF2 requirements gathering. I tried to play the IRC
meeting agenda game and added my in-contention modular items to the
meeting agenda -for discussion-.

Zend's leadership (which might be you or those above, I don't know)
clearly does not understand how to manage community development.
Community development is not where we all go to our silo's and when its
all done come back and slap it down as a ZFPROP or RFC and say 'done,
review, here's benchmarks' -- but that is the crux of the issue today.

Getting the ZF2 architecture right will require actual, genuine
collaboration by Zend and the community right from the conceptual ideas.
At this point, a decision has been made to end debate on those ideas and
move on to your vision (in what I'd consider a very undemocratic and
exceptionally broken process).

I'm committed to staying out of the frameworks' development from now on,
because obviously this back and forth isn't working, however, please
refrain from saying I didn't put forth an alternative proposal. I
certainly did. I've devoted countless unpaid hours to this mailing list,
the framework discussions and architecture -- I even wrote a book
evangelising the framework and its community. You cannot lay the blame
on me.

I tried my very best to bring community development to this framework,
to communicate my ideas and concepts clearly, and to play the ridiculous
and ever-changing process game. That you've now lost yet another one of
the frameworks biggest supporters is fully your responsibility.

--

Kevin McArthur

On 11-09-01 08:58 AM, Matthew Weier O'Phinney wrote:

> -- Artur Bodera<[hidden email]>  wrote
> (on Thursday, 01 September 2011, 11:59 AM +0200):
>> Because I've been pointed at (along with Kevin) during last IRC
>> meeting, I'd like to continue a very important IMO discussion on one
>> topic.
>>
>> I've spent some time with Kevin on IRC later that night because I
>> wanted to get a better glimpse on what was his perception on problems
>> and scale of problems that, in his opinion, ZF project has.
>>
>> Today, I'll try to bring up the first challenge up to discussion.
>>
>> I'm also personally unsatisfied by how the "module vs component"
>> discussion on irc has been muted, and how Matthew flattened it by
>> pointing at us as the "only 2 people that had problem with the
>> definition". Fortunately here, I have a lot more space to explain in
>> more detail, what many of you are missing out in this discussion and
>> there is no one to "vote me out" ;-P
> I'm sorry you feel you were muted. The point I was trying to make during
> the meeting is that the meeting is for fleshing out final details and
> making decisions. New ideas should be proposed via RFCs and dedicated ML
> threads.
>
> Several of us did exactly that in the last few weeks and threw out
> proposals for how we define these concepts, and how we might move
> forward. While you and Kevin both participated in these discussions,
> neither of you put forth an alternate proposal. As such, all any of us
> on the ML have is a piece-meal view of what either of you envision --
> which makes discussion during the meetings very, very difficult. We have
> nothing concrete to look at.
>
> In the mean time, there _were_ concrete RFCs posted that we _could_
> discuss and vote on.
>
> Now, to continue: thank you for this post, as it finally puts forth a
> more concrete vision to discuss. More below.
>
> <snip>
>
>> The most important message I'd like to bring up today is the
>> difference between a "modular architecture" of a project and an
>> "architecture for modules".
> <snip>
>
>> Modules: I personally use mvc modules in ZF1 since the dawn of
>> Zend_Application. Even though they didn't get much lovin and are a
>> mess (i.e. boostrap re-loads/double-loads the same resources). There
>> has been some evolution in the thinking, as now "modules"  can have
>> assets, configuration, routes, etc. and these will be nicely,
>> automatically handled, merged and resolved by MVC+DI.
>>
>> This is what you call "and architecture for modules". ZF2 will have
>> more tools than ZF1 to handle installation of blobs of code, images
>> and other stuff. It will be able to do tasks I've mentioned. ZF2 will
>> be a foundation to build MVC modules on.
>>
>> Kevin named ZF2:  "ZF1 with modules"
>>
>> It is more hurtful than you might think.
>>
>>
>> What is a modular architecture then?
>>
>> Notice how I previously wrote "modular architecture for a PROJECT".
>> What I mean by that, is that modular philosophy crosses and shoots
>> through all levels of A PROJECT. What does that mean in practice?
>>
>>   * flexibility
>>      Everything is optional. MVC is optional, Ralph's DB is optional,
>>      Form is optional, Tool, Autoloading and Application is optional (I
>>      know what I'm poking :) ).
> This is already what we're planning, though. If you want to use a single
> component, you grab it via the installer (whatever we end up choosing).
> Want just Zend\Db?
>
>      prompt>  <installer>  install zf\Zend_Db
>
> Does this differ from what you mean/intend? If so, how?
>
>>   * extendability and decoupling
>>      There is no single component, or there are very few that cannot be
>>      extended and/or replaced with a custom implementation. All
>>      components are equal in rights, ZF-reviewed or not.  All are
>>      obtainable and distributed the same way.
> Again, this is also a plan. One guiding principle for ZF2 has been that
> we try and follow SOLID principles when possible, and define interfaces
> for anything we think could provide an extension point -- which is a lot
> more often than not. Does this differ from what you mean/intend? If so,
> how?
>
> Regarding all components being equal, "ZF-reviewed or not," I don't know
> that we can do that while we require a CLA, nor from a maintenance
> perspective. One big point of the proposal process is to identify
> components that we think will be of enough significant general use that
> we can commit to maintaining them long-term. Simply put, if there's not
> enough interest to maintain a component, we shouldn't be shipping it.
>
> That said, I _do_ think components should be obtainable and distributed
> in a consistent manner, whether or not they are an official part of the
> ZF distribution, or part of the larger ZF user/developer ecosystem.
> That's where finding a sane and powerful installation tool comes in;
> any tool we choose should allow for registering channels or repository
> sources in a central location to allow developers to search for
> components to use. Does this differ from what you intend? If so, how?
>
>>   * consistency
>>      Because of obvious marketing and educational reasons, there should
>>      be a "popular", "standard" distribution of a framework, but at
>>      architectural level (with everything optional) all pieces of the
>>      puzzle are replaceable. ZF2 standard distribution becomes an
>>      example of how thin the ZF2 core is and how any modules/components
>>      can be bundled even if they don't have to be.
> This was discussed in-depth yesterday during the distribution portion of
> the meeting, and I've updated the distribution RFC based on that
> discussion. I personally feel that:
>
>   * Providing an installable package per component, plus
>   * A variety of meta-packages describing groupings of components
>
> answers this well. Based on the IRC discussion yesterday, you seemed to
> agree. Does this differ from what you intend? If so, how?
>
>>   * openness
>>      Openness (h@ck4bility) is a door to innovation. A community
>>      project with modular architecture invites people to write their
>>      stuff. Most of it will be crappy in quality, many will be
>>      worthwhile a few will shine.
>>
>> There is an obvious conflict between code quality standards, legal
>> issues (CLA) and that openness. I've had bricks thrown at my by Paddy
>> several times when mentioning of having 3rd party components in the
>> Zend\* namespace, as if the PHP 5.3 NS now became the international
>> catalog of brand's code quality ranking something.
> It's not just about quality; it's about what is considered code that the
> ZF project _MAINTAINS_ and _SUPPORTS_. Having 3rd-party components share
> the same namespace makes it difficult for end-users to know who to go to
> for help, to report issues, or to submit patches.
>
> Openness is fine, and I fully support having a community hub for modules
> -- but we simply can't have folks using the "Zend" namespace for code
> we're not providing directly from the ZF project repository, for the
> reasons outlined in the previous paragraph.
>
> The idea of a forge site for 3rd-party code has been raised a ton, and a
> thread or three have even been started on this ML amongst folks
> interested in building such a site. I'd love to see one; I also know
> that my team has too-limited resources to do it while also working on
> ZF2 and helping maintain ZF1. It has to be a community initiative at
> this point; however, once something workable is in place, we can
> certainly help support it in terms of providing a home (server and DNS)
> for it, as well as access for maintainers to assist.
>
>> I am a developer who values good unit tests, stability and license
>> transparency.  On the other hand, I cannot overlook what recent years
>> have shown us with other agile projects. Dev. community has noticed,
>> that they have a predictable probability distribution in terms of code
>> quality. Although I don't have solid data for that, from my experience
>> projects like drupal, jquery or even pear, comply to the 90-9-1
>> theory.
>> (http://www.wikipatterns.com/display/wikipatterns/90-9-1+Theory)
>>
>> This means that after we create (which has been "voted over" :P) a
>> marketplace, catalog for 3rd party components, 90% will potentially be
>> crap, 9% will be very popular, high-quality functional components that
>> will shape the perceived value of the ecosystem, and 1% will be "gems"
>> - hero components that will be a presented on keynotes and will build
>> the brand for the project.
> And I'd argue that the "gems" and some of the "popular, high-quality"
> components become candidates for inclusion in the main project itself.
>
> I don't see any conflict here, to be honest. With DVCS, it becomes
> trivial to merge such components into the master repository over time --
> while allowing the projects to maintain independence if desired, but
> also have independence while they work out a variety of kinks on a more
> agile basis before they suggest inclusion in the main project.
>
>> I'm also a man of action. I'm not a fan of ranting just to point out
>> what's broken.  I prefer to have a concept, solution proposition
>> before I step out. Here is what I propose:
>>
>> 1. THIN CORE
>> In order to meet the need of flexibility, we need a stable but very
>> thin core. This means a collection of "components" that are foundation
>> for other components.  Core can also maintain and handle any number of
>> additional modules/components.
> This is planned. Do you disagree with the core as defined in the
> distribution RFC? If so, what changes would you make?
>
>> 2. CONSISTENCY AND SIMPLIFICATION
>> Review the concept of "module == component". This will be particularly
>> painful for all purists, but please try to look through it. It's
>> (again) not about naming conventions but about __architecture__ A
>> "module == component" is about having a common name for the concept of
>> reusable code. Here is an example
>>
>>   * MVC module
>>   * Security module
>>   * Zend\Db\Adapter\Firebird module
>>   * Zend\Form\Element\Video module (or VideoElement module)
>>
>> The goal of this approach is simplify.
>> instead of:
>>      "framework consisting of components, assets, meta-modules, packages,
>> adapters, ...."
>> think about:
>>      "framework consisting of number of modules",
> I actually find the above _more_ confusing, to be honest. Why wouldn't
> all form-related classes be in a single module? and DB adapters? I could
> understand if they are _third-party_ offerings, but within the framework
> itself?
>
> Also, you've misquoted. The framework does not consist of "components,
> assets, meta-modules, packages, adapters, ...." That has never been
> stated. The framework as _currently_ proposed, consists of components --
> code, tests, and documentation only.
>
> _Distribution_ would allow selecting individual components and/or
> meta-packages consisting of multiple components. That's what the
> Distribution RFC was trying to identify.
>
> I think part of the problem I'm having is that I'm not seeing any
> concrete details on what a module, as you conceive it, _looks_ like, in
> the repository, nor how this impacts maintenance and packaging.
>
> In the current paradigm, running tests is easy -- a single directory
> contains tests for the entire framework, making it easy to offer a
> single autoloader for loading all library and test assets during
> testing. Because of the flexibility of PHPUnit, we can easily test a
> single component or class, or use annotations to group components and/or
> tests for running at once.
>
> Building the manual is also similarly easy: a single manual.xml file is
> used, and pulls in all manual pages via xincludes. Translation is fairly
> easy to accomplish, as we can copy the english sources into each
> translation directory, making it easy to run statistics on translation
> coverage. Fallbacks can be defined for xincludes to allow falling back
> to english if a translation is missing.
>
> If I understand correctly, you're positing that each component becomes a
> self-contained module, each containing code, tests, and documentation:
>
>      Zend-Form/
>          library/
>              Zend/
>                  Form/
>                      ....
>          tests/
>              Zend/
>                  Form/
>                      ...
>          documentation/
>              manual.xml
>              specs/
>                  ...
>
> This has huge ramifications in terms of how the repository is
> structured, how tests are run, and how the manual is built.
>
> On the plus side, the entire component is in one place, and it's easy to
> visualize what a package looks like for a discrete component. It would
> play very easily into any packaging system and distribution tool we
> might select.
>
> On the flip side:
>
>   * How does the component know where dependencies exist?
>      * This is important both when installed, as well as when *running*
>        *the* *tests*.
>   * How does documentation work?
>      * On the website, we'd want a single tree of manuals in order to
>        promote easy discovery. This implies a single build system -- how
>        would we accomplish that?
>      * Within a single module, do we build an individual manual for that
>        module for distribution? What about linking to other component
>        documentation -- how are those links resolved, particularly if
>        that component is not installed?
>
> Another point that comes to mind: you show individual adapters as being
> potentially modules -- why? Wouldn't that make things _more_ confusing?
> Shouldn't grabbing a component also grab all subcomponents such as
> adapters -- and why wouldn't we want these as a single package?
>
> I can _definitely_ see the pros to such an architecture -- don't get me
> wrong. I just also see a ton of HUGE infrastructure issues that would
> need to be addressed, and I think we have a lot of work to iron those
> out if we want to move forward. Frankly, I'm not sure anybody wants new
> development lingering for weeks or months before we figure those out,
> when the current infrastructure already is tried, tested, and proven.
>
> Feel free to try and convince me otherwise, though -- as I said, I can
> see the benefits; I'm just not sure they outweight the costs of change
> currently.
>
>> "Everything except for core is a module" will also meet the
>> requirement of openness and consistency. Because both zf-contributors
>> modules and "3rd party" modules are on thesame page, it is a truly
>> modular architecture. Before bashing this argument, please take a
>> moment to browse through Drupal and Wordpress. Because these are
>> CMSes, their core is not thin by ZF-standards, but it is ridiculously
>> thin by CMS standards. And, everything in these projects is a drop-in
>> module (also, peek again at the sheer number of modules available for
>> download for these projects)
>>
>> Consistent naming convention will greatly help with education.
>> Instead of saying:
>>      - grab the FormElements module, then download Captcha component
>>        and then look for ReCaptcha adapter which goes to xyz folder"
>>
>> we'll just say:
>>      - install ReCaptcha module
> Um, I think you've convoluted the story significantly, and not really
> detailed what you think you're trying to install here.
>
> If you're wanting to install the ReCaptcha form element, I don't see why
> it would be different than this:
>
>      prompt>  <installer>  install zf\Zend_Form
>      prompt>  <installer>  install zf\Zend_Service_ReCaptcha
>
> I think this would be true even in the modular architecture you envision
> -- the recaptcha form element consumes the recaptcha service as an
> optional dependency (if missing, it obviously fails all over the place,
> but you can still install the form component without it); the recaptcha
> service can be used without the form component.
>
> Can you provide a different example that better illustrates the
> differences in what has been proposed vs what you envision, please?
>
>> 3. THERE IS NO 3RD PARTY.  (also "Everything is miscellaneous").
>> To meet the requirement of openness, there can be no wall between us
>> (????), contributors and other people who just had an account set up
>> on github, or email at gmail. Contributing code is not the same as
>> writing stupid comments on facebook or uploading cat photos on
>> icanhazcheeseburger. It takes more knowledge and time, so there will
>> be less "spam". In terms of code quality, this is what GUIDELINES are
>> for.  We can have both! Openness and code quality guidelines.  We
>> _must not_ have a distinction between "us" and "them". There is no
>> "them".  We _must not_ have a wall of review process. The market does
>> that for us.  People will do that for us.
> I've outlined above why I think we need to make a differentiation
> between the official ZF distribution (the framework upon which
> applications are built) and a community forge/hub/etc. I'm not going
> over that again here.
>
>> I'm sorry, but trying to say that each and every ZF1 class is an
>> example of engineering brilliance is a huge overstatement ;)
> ZF1 has evolved tremendously over time. Many of the testing and
> documentation requirements also evolved -- most of these were missing
> prior to the 1.0 release, meaning that older code often is not
> engineered to the same standards that recent contributions are.
>
> Additionally, the PHP landscape has changed a ton in the last 5.5 years
> since the initial 0.1.0 release -- understanding and application of
> patterns, testing practices, etc. have taken a quantum leap forward
> since then. Yes, some code in ZF1 is better than others. The point is
> that we have standards that we've come to expect that we can now apply
> to the code base as a whole for ZF2.
>
>> How does it translate to real world? Well, some people just don't use
>> Zend_Db or Zend_Form, or Zend_Mail.  Same goes with drupal. Many
>> "standard modules" have been replaced by higher quality modules from
>> the catalog, from "the community". Now they are part of core or are
>> bundled in standard distribution.
> Having use-at-will installation answers your first question. Regarding
> the second part of your paragraph, that's what having a new major
> version allows us to do -- get rid of cruft and deprecated
> functionality, and add new code.
>
>> Modular architecture in this regard means, that each we give people a
>> choice of which and whose module I'll use for forms, and db and
>> mailing people.
> I see no conflict here. Assuming an installer tool that speaks to a
> server allowing developers to register their own repositories,
> developers can search for and install any component they want, be it
> from the official Zend repo or otherwise.
>
>> What about current zend-review components? Put them inside
>> repository/catalog ALONG with all other components. Because the CR
>> team has put a lot of effort and their time, we can have a
>> [zend-certified] badge or some other form of letting people know that
>> these components are _different_.  NOT "better"... "different".  You
>> can say "featured" for a lack of better word at this time.
>>
>> I'm sorry, but assuming that ZF review process somewhat makes ZF
>> components better than any external person in the world would write
>> them is just mathematically nearly impossible :-) I'm against
>> discrimination which will discourage any contributors that might just
>> find a better way to solve XYZ problem that the "almighty
>> zf-contributor ML subscribers".
> That's not the argument for inclusion in the ZF distribution AT ALL! The
> argument for inclusion is CAN WE and WILL WE SUPPORT and MAINTAIN the
> component long-term? I.e., is the component something that has a
> demonstrated need, and for which we will be able to find new maintainers
> should the original leave in the future?
>
> An official repo can easily live alongside a community repo -- or
> distributed individual repos. We can setup infrastructure that allows
> folks to search for functionality they desire from their installer, and
> which searches all registered repos. In that situation, the ZF repo is
> simply _one_ _source_ -- one that many modules/components/whatever might
> list dependencies against, but simply one source. This allows somebody
> to distribute their own form component, or DB implementation, or ORM, or
> whatever, and have it installed directly in their app or alongside the
> ZF installation (or set of ZF components they have installed).
>
>> If ZF2 became an open, modular, component framework it will create an
>> ecosystem.
> Yes, we need an ecosystem. I argue the way to do that is to be smart
> about selecting a packaging and installation tool that allows us to
> aggregate repos from disparate sources, including the ZF repo.
>
> The current processes are in place to create a story for maintenance and
> longevity of the code shipped directly with the framework. How
> individuals version and maintain their own repositories is up to them;
> we can simply make it easy for them to get their code in the hands of
> users.
>

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Modular architecture != architecture for modules

weierophinney
Administrator
-- Kevin McArthur <[hidden email]> wrote
(on Thursday, 01 September 2011, 09:30 AM -0700):

> I'll leave the rest of the discussion to you folks and the
> community, but I have to respond to one specific statement:
>
> While you and Kevin both participated in these discussions,
> neither of you put forth an alternate proposal.
>
> That's absolutely not true. I've penned dozens (potentially
> hundreds) of emails to this list on the concepts I envision. I've
> contributed significantly to the ZF2 requirements gathering. I tried
> to play the IRC meeting agenda game and added my in-contention
> modular items to the meeting agenda -for discussion-.

Yes, you've penned many emails. The problem is that none of them
describes the vision you have completely; it's up to those of us reading
the emails to go through and try to piece it together from various
threads.

What I was getting at is that others of us have posted either RFCs or
started threads that detail something concrete, and then asked for
feedback. This gives a _starting_ point that everyone can use in
discussion. If a competing idea crops up, then the person originating it
should start a new RFC or thread proposing it.

I'm going to be honest -- you've brought up some great ideas during the
various discussions, and I _have_ incorporated a number of the ideas in
revisions to RFCs and threads based on your ideas. But when you go into
the meeting and start bashing the ideas under discussion without having
something concrete to point to, how can we have any real, fruitful,
discussion?

When I say "concrete", I don't mean a fully-developed, fully working
system, either. I mean something like:

 * Definitions. "A component is...," "a module is...," even, "there
   should be no difference between a module and component." Basically,
   establishing a shared vocabulary. If the definitions differ from
   common conceptions shared in ZF1, I think there needs to be a good
   justification.
 
 * Directory structures -- how is code and/or non-code organized? How
   will it impact development and maintenance? What new infrastructure
   might we need? what infrastructure would we be able to get rid of?
   etc.

 * Pseudo-code: how do you envision using the code and/or system?

These give us an idea of what you envision. It's something we can point
at -- "item X on page such-and-such states ... I think ...". As others
have pointed out in OSS development, code rules -- and I think that
applies to pseudo-code as well. We're developers; we like to have
something to look at to see how things might work.

That's all I'm trying to get at. You can't get buy-in for an idea unless
you can express it in one place -- making us try and put it together
from dozens of emails makes it too easy to misinterpret intent.

> Zend's leadership (which might be you or those above, I don't know)
> clearly does not understand how to manage community development.
> Community development is not where we all go to our silo's and when
> its all done come back and slap it down as a ZFPROP or RFC and say
> 'done, review, here's benchmarks' -- but that is the crux of the
> issue today.

I'm sorry you feel that way. Yes, we've messed up with regards to the
process. We're trying to change that now. Change takes time, and it's
been but a few weeks since we've listened to the calls for action by the
community and started taking steps to open up.

The prototype I'll be pushing into a branch later will be just that -- a
prototype. I'm completely open for folks to say it's no good -- but at
least it gives us a place to _begin_ discussions, something concrete. Is
it the result of a lot of development? Sure. Does that mean it's final?
No.

As I said above, I don't think RFCs or proposals need to be final. What
they _should_ do, however, is provide a solid foundation to discuss one
or more discrete ideas so that we can gather feedback and consensus --
even if that consensus is "ditch it."

> Getting the ZF2 architecture right will require actual, genuine
> collaboration by Zend and the community right from the conceptual
> ideas. At this point, a decision has been made to end debate on
> those ideas and move on to your vision (in what I'd consider a very
> undemocratic and exceptionally broken process).

Again, I'm sorry you feel this way. I think that there's far more
collaboration going on now than we've had all year, and I think we're
striving to be more democratic. Again, imperfect system, but we're
trying to improve.

I sincerely hope you'll keep an eye on development as it progresses, and
that you see something that changes your mind about contributing.

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Modular architecture != architecture for modules

Ralph Schindler-2
In reply to this post by Kevin McArthur-2
Kevin,

> That's absolutely not true. I've penned dozens (potentially hundreds) of
> emails to this list on the concepts I envision. I've contributed
> significantly to the ZF2 requirements gathering. I tried to play the IRC
> meeting agenda game and added my in-contention modular items to the
> meeting agenda -for discussion-.

I've read the emails. And yes, that is a completely valid form of
engagement.  But when people fail to see/understand your vision in your
email, it's time to write code.  Where is the github account? Where is
the fork of zf2 with your ideas in place?  Where are the prototypes of
your ideas?

More on this down below.

I really try to get all of your concepts, but most times we are blending
implementation details with overall architectural requirements and it
becomes next to impossible to come to any conclusion.

I take offense that in recent emails you reference Zend\Di, and the
performance implications when in fact it's clear you've not completely
played with the code or offered patches/ideas for improvement etc. (You
reference there is no Di compiler when in fact its been in place for
over 3 months).

> Getting the ZF2 architecture right will require actual, genuine
> collaboration by Zend and the community right from the conceptual ideas.
> At this point, a decision has been made to end debate on those ideas and
> move on to your vision (in what I'd consider a very undemocratic and
> exceptionally broken process).

Open source is never a democracy.  At best, it's a meritocracy ... and
usually, there are a few stakeholder that get priority over others.
That's just the way it is.  If you don't like someones vision of how
things work, you need to convince not only them but a few others that
not only is that someone else's vision is wrong but that your vision is
better.  It takes work.  A single email usually doesn't do the trick,
especially in an open source project where everyone is generally pretty
passionate about their current coding ideology/religion.

> I tried my very best to bring community development to this framework,
> to communicate my ideas and concepts clearly, and to play the ridiculous
> and ever-changing process game. That you've now lost yet another one of
> the frameworks biggest supporters is fully your responsibility.

That's not a fair statement.  Your ideas and concepts are not clear, we
need code and prototypes.

We're not even asking for tests and documentation that you claim you
can't be bothered to write because your clients can't be asked to pay
for it.  Which, by the way, is NOT fair to the contributors that have
swallowed the "dreaded" documentation and tests pill that comes along
with being part of the community contribution process.

-ralph



--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Modular architecture != architecture for modules

Kevin McArthur-2
Ralph,

I'm sorry to get you involved in this, because I really do respect your
development and talent.

> I've read the emails. And yes, that is a completely valid form of
> engagement.  But when people fail to see/understand your vision in
> your email, it's time to write code.  Where is the github account?
> Where is the fork of zf2 with your ideas in place?  Where are the
> prototypes of your ideas?

This is the problem here. It presumes a one-man-show approach to
development. I could write code for specific pieces, but to code the
whole thing would take months (as Zend is experiencing in getting out
the prototypes from their end). I'm not paid to work on the Zend
Framework. I can carve out maybe 5 hours a week to work on open-source
projects for fun. I simply don't have the bandwidth to be a one-man-show
here.

This is what I mean when I say the community development is broken
coming from Zend. Why is Zend not trying to incorporate the community
into it's team? Why is it 'heres a prototype' rather than 'help me build
a prototype'. This approach ensures that since there's a significant
investment of time and effort and that decisions are exceptionally hard
to back out of.

My concepts require big picture thinking outside the ZF1-legacy box. I
asked for that input on the mailing list, in the rfcs, in the
requirements gathering, in the irc meeting agenda. It lead no where and
somehow defacto moved on to the Zend approach which I will liken to
ZF1+modules. So fine, its the Zend framework after-all, but a one-way
product from Zend is not something I can contribute meaningfully to.

>
> Open source is never a democracy.  At best, it's a meritocracy ... and
> usually, there are a few stakeholder that get priority over others.
> That's just the way it is.  If you don't like someones vision of how
> things work, you need to convince not only them but a few others that
> not only is that someone else's vision is wrong but that your vision
> is better.  It takes work.  A single email usually doesn't do the
> trick, especially in an open source project where everyone is
> generally pretty passionate about their current coding ideology/religion.

I have been working on the architecture questions since the 1.0 freeze
was imposed. Collaborations with dozens of contributors, forks of code,
and an ungodly amount emails. This fight has been ongoing for the better
part of 5 years. It finally comes time to hash it out and build 2.0, and
its 30 minutes, this isnt getting anywhere and lets move on to the Zend way.

>
>> I tried my very best to bring community development to this framework,
>> to communicate my ideas and concepts clearly, and to play the ridiculous
>> and ever-changing process game. That you've now lost yet another one of
>> the frameworks biggest supporters is fully your responsibility.
>
> That's not a fair statement.  Your ideas and concepts are not clear,
> we need code and prototypes.

I've provided clarification every time its been asked for. I don't have
code or prototypes nor the resources required to create them myself. I
need a collaborative community of developers to make my vision a
reality. I cannot push a prototype on the community for review.

It is a fair statement from my perspective.

>
> We're not even asking for tests and documentation that you claim you
> can't be bothered to write because your clients can't be asked to pay
> for it.  Which, by the way, is NOT fair to the contributors that have
> swallowed the "dreaded" documentation and tests pill that comes along
> with being part of the community contribution process.

Its also not fair to the users of the framework who have not benefited
from significant contributions due to process overhead and contributor
attrition. Its hard to quantify what doesn't get made, but I'm certain
its significant.

I am sorry that we couldn't work it out, especially because I do
remember a day when you and I, Ralph, would sit down and hack through
code and architecture together and create awesome additions to the
framework. The collaboration on the initial Zend_Tool was awesome for
example, so I really do hope you guys get this sorted out,

--

Kevin

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|

RE: Modular architecture != architecture for modules

The Evil Thug
In reply to this post by Kevin McArthur-2
> Getting the ZF2 architecture right will require actual, genuine
> collaboration by Zend and the community right from the conceptual ideas.
> At this point, a decision has been made to end debate on those ideas and
> move on to your vision (in what I'd consider a very undemocratic and
> exceptionally broken process).

After YEARS of conceptualizing professor, don't you think it's about time
for the rubber to hit the road and to actually produce SOMETHING?

Too many Chiefs, not enough Braves.

Elegant design is certainly nice, but a global assembly cache never produced
doesn't beat an ini file designed and produced in a week. Sorry.

-----Original Message-----
From: Kevin McArthur [mailto:[hidden email]]
Sent: Thursday, September 01, 2011 12:31 PM
To: [hidden email]
Subject: Re: [zf-contributors] Modular architecture != architecture for
modules

Matthew,

I'll leave the rest of the discussion to you folks and the community,
but I have to respond to one specific statement:

While you and Kevin both participated in these discussions,
neither of you put forth an alternate proposal.

That's absolutely not true. I've penned dozens (potentially hundreds) of
emails to this list on the concepts I envision. I've contributed
significantly to the ZF2 requirements gathering. I tried to play the IRC
meeting agenda game and added my in-contention modular items to the
meeting agenda -for discussion-.

Zend's leadership (which might be you or those above, I don't know)
clearly does not understand how to manage community development.
Community development is not where we all go to our silo's and when its
all done come back and slap it down as a ZFPROP or RFC and say 'done,
review, here's benchmarks' -- but that is the crux of the issue today.

Getting the ZF2 architecture right will require actual, genuine
collaboration by Zend and the community right from the conceptual ideas.
At this point, a decision has been made to end debate on those ideas and
move on to your vision (in what I'd consider a very undemocratic and
exceptionally broken process).

I'm committed to staying out of the frameworks' development from now on,
because obviously this back and forth isn't working, however, please
refrain from saying I didn't put forth an alternative proposal. I
certainly did. I've devoted countless unpaid hours to this mailing list,
the framework discussions and architecture -- I even wrote a book
evangelising the framework and its community. You cannot lay the blame
on me.

I tried my very best to bring community development to this framework,
to communicate my ideas and concepts clearly, and to play the ridiculous
and ever-changing process game. That you've now lost yet another one of
the frameworks biggest supporters is fully your responsibility.

--

Kevin McArthur

On 11-09-01 08:58 AM, Matthew Weier O'Phinney wrote:

> -- Artur Bodera<[hidden email]>  wrote
> (on Thursday, 01 September 2011, 11:59 AM +0200):
>> Because I've been pointed at (along with Kevin) during last IRC
>> meeting, I'd like to continue a very important IMO discussion on one
>> topic.
>>
>> I've spent some time with Kevin on IRC later that night because I
>> wanted to get a better glimpse on what was his perception on problems
>> and scale of problems that, in his opinion, ZF project has.
>>
>> Today, I'll try to bring up the first challenge up to discussion.
>>
>> I'm also personally unsatisfied by how the "module vs component"
>> discussion on irc has been muted, and how Matthew flattened it by
>> pointing at us as the "only 2 people that had problem with the
>> definition". Fortunately here, I have a lot more space to explain in
>> more detail, what many of you are missing out in this discussion and
>> there is no one to "vote me out" ;-P
> I'm sorry you feel you were muted. The point I was trying to make during
> the meeting is that the meeting is for fleshing out final details and
> making decisions. New ideas should be proposed via RFCs and dedicated ML
> threads.
>
> Several of us did exactly that in the last few weeks and threw out
> proposals for how we define these concepts, and how we might move
> forward. While you and Kevin both participated in these discussions,
> neither of you put forth an alternate proposal. As such, all any of us
> on the ML have is a piece-meal view of what either of you envision --
> which makes discussion during the meetings very, very difficult. We have
> nothing concrete to look at.
>
> In the mean time, there _were_ concrete RFCs posted that we _could_
> discuss and vote on.
>
> Now, to continue: thank you for this post, as it finally puts forth a
> more concrete vision to discuss. More below.
>
> <snip>
>
>> The most important message I'd like to bring up today is the
>> difference between a "modular architecture" of a project and an
>> "architecture for modules".
> <snip>
>
>> Modules: I personally use mvc modules in ZF1 since the dawn of
>> Zend_Application. Even though they didn't get much lovin and are a
>> mess (i.e. boostrap re-loads/double-loads the same resources). There
>> has been some evolution in the thinking, as now "modules"  can have
>> assets, configuration, routes, etc. and these will be nicely,
>> automatically handled, merged and resolved by MVC+DI.
>>
>> This is what you call "and architecture for modules". ZF2 will have
>> more tools than ZF1 to handle installation of blobs of code, images
>> and other stuff. It will be able to do tasks I've mentioned. ZF2 will
>> be a foundation to build MVC modules on.
>>
>> Kevin named ZF2:  "ZF1 with modules"
>>
>> It is more hurtful than you might think.
>>
>>
>> What is a modular architecture then?
>>
>> Notice how I previously wrote "modular architecture for a PROJECT".
>> What I mean by that, is that modular philosophy crosses and shoots
>> through all levels of A PROJECT. What does that mean in practice?
>>
>>   * flexibility
>>      Everything is optional. MVC is optional, Ralph's DB is optional,
>>      Form is optional, Tool, Autoloading and Application is optional (I
>>      know what I'm poking :) ).
> This is already what we're planning, though. If you want to use a single
> component, you grab it via the installer (whatever we end up choosing).
> Want just Zend\Db?
>
>      prompt>  <installer>  install zf\Zend_Db
>
> Does this differ from what you mean/intend? If so, how?
>
>>   * extendability and decoupling
>>      There is no single component, or there are very few that cannot be
>>      extended and/or replaced with a custom implementation. All
>>      components are equal in rights, ZF-reviewed or not.  All are
>>      obtainable and distributed the same way.
> Again, this is also a plan. One guiding principle for ZF2 has been that
> we try and follow SOLID principles when possible, and define interfaces
> for anything we think could provide an extension point -- which is a lot
> more often than not. Does this differ from what you mean/intend? If so,
> how?
>
> Regarding all components being equal, "ZF-reviewed or not," I don't know
> that we can do that while we require a CLA, nor from a maintenance
> perspective. One big point of the proposal process is to identify
> components that we think will be of enough significant general use that
> we can commit to maintaining them long-term. Simply put, if there's not
> enough interest to maintain a component, we shouldn't be shipping it.
>
> That said, I _do_ think components should be obtainable and distributed
> in a consistent manner, whether or not they are an official part of the
> ZF distribution, or part of the larger ZF user/developer ecosystem.
> That's where finding a sane and powerful installation tool comes in;
> any tool we choose should allow for registering channels or repository
> sources in a central location to allow developers to search for
> components to use. Does this differ from what you intend? If so, how?
>
>>   * consistency
>>      Because of obvious marketing and educational reasons, there should
>>      be a "popular", "standard" distribution of a framework, but at
>>      architectural level (with everything optional) all pieces of the
>>      puzzle are replaceable. ZF2 standard distribution becomes an
>>      example of how thin the ZF2 core is and how any modules/components
>>      can be bundled even if they don't have to be.
> This was discussed in-depth yesterday during the distribution portion of
> the meeting, and I've updated the distribution RFC based on that
> discussion. I personally feel that:
>
>   * Providing an installable package per component, plus
>   * A variety of meta-packages describing groupings of components
>
> answers this well. Based on the IRC discussion yesterday, you seemed to
> agree. Does this differ from what you intend? If so, how?
>
>>   * openness
>>      Openness (h@ck4bility) is a door to innovation. A community
>>      project with modular architecture invites people to write their
>>      stuff. Most of it will be crappy in quality, many will be
>>      worthwhile a few will shine.
>>
>> There is an obvious conflict between code quality standards, legal
>> issues (CLA) and that openness. I've had bricks thrown at my by Paddy
>> several times when mentioning of having 3rd party components in the
>> Zend\* namespace, as if the PHP 5.3 NS now became the international
>> catalog of brand's code quality ranking something.
> It's not just about quality; it's about what is considered code that the
> ZF project _MAINTAINS_ and _SUPPORTS_. Having 3rd-party components share
> the same namespace makes it difficult for end-users to know who to go to
> for help, to report issues, or to submit patches.
>
> Openness is fine, and I fully support having a community hub for modules
> -- but we simply can't have folks using the "Zend" namespace for code
> we're not providing directly from the ZF project repository, for the
> reasons outlined in the previous paragraph.
>
> The idea of a forge site for 3rd-party code has been raised a ton, and a
> thread or three have even been started on this ML amongst folks
> interested in building such a site. I'd love to see one; I also know
> that my team has too-limited resources to do it while also working on
> ZF2 and helping maintain ZF1. It has to be a community initiative at
> this point; however, once something workable is in place, we can
> certainly help support it in terms of providing a home (server and DNS)
> for it, as well as access for maintainers to assist.
>
>> I am a developer who values good unit tests, stability and license
>> transparency.  On the other hand, I cannot overlook what recent years
>> have shown us with other agile projects. Dev. community has noticed,
>> that they have a predictable probability distribution in terms of code
>> quality. Although I don't have solid data for that, from my experience
>> projects like drupal, jquery or even pear, comply to the 90-9-1
>> theory.
>> (http://www.wikipatterns.com/display/wikipatterns/90-9-1+Theory)
>>
>> This means that after we create (which has been "voted over" :P) a
>> marketplace, catalog for 3rd party components, 90% will potentially be
>> crap, 9% will be very popular, high-quality functional components that
>> will shape the perceived value of the ecosystem, and 1% will be "gems"
>> - hero components that will be a presented on keynotes and will build
>> the brand for the project.
> And I'd argue that the "gems" and some of the "popular, high-quality"
> components become candidates for inclusion in the main project itself.
>
> I don't see any conflict here, to be honest. With DVCS, it becomes
> trivial to merge such components into the master repository over time --
> while allowing the projects to maintain independence if desired, but
> also have independence while they work out a variety of kinks on a more
> agile basis before they suggest inclusion in the main project.
>
>> I'm also a man of action. I'm not a fan of ranting just to point out
>> what's broken.  I prefer to have a concept, solution proposition
>> before I step out. Here is what I propose:
>>
>> 1. THIN CORE
>> In order to meet the need of flexibility, we need a stable but very
>> thin core. This means a collection of "components" that are foundation
>> for other components.  Core can also maintain and handle any number of
>> additional modules/components.
> This is planned. Do you disagree with the core as defined in the
> distribution RFC? If so, what changes would you make?
>
>> 2. CONSISTENCY AND SIMPLIFICATION
>> Review the concept of "module == component". This will be particularly
>> painful for all purists, but please try to look through it. It's
>> (again) not about naming conventions but about __architecture__ A
>> "module == component" is about having a common name for the concept of
>> reusable code. Here is an example
>>
>>   * MVC module
>>   * Security module
>>   * Zend\Db\Adapter\Firebird module
>>   * Zend\Form\Element\Video module (or VideoElement module)
>>
>> The goal of this approach is simplify.
>> instead of:
>>      "framework consisting of components, assets, meta-modules, packages,
>> adapters, ...."
>> think about:
>>      "framework consisting of number of modules",
> I actually find the above _more_ confusing, to be honest. Why wouldn't
> all form-related classes be in a single module? and DB adapters? I could
> understand if they are _third-party_ offerings, but within the framework
> itself?
>
> Also, you've misquoted. The framework does not consist of "components,
> assets, meta-modules, packages, adapters, ...." That has never been
> stated. The framework as _currently_ proposed, consists of components --
> code, tests, and documentation only.
>
> _Distribution_ would allow selecting individual components and/or
> meta-packages consisting of multiple components. That's what the
> Distribution RFC was trying to identify.
>
> I think part of the problem I'm having is that I'm not seeing any
> concrete details on what a module, as you conceive it, _looks_ like, in
> the repository, nor how this impacts maintenance and packaging.
>
> In the current paradigm, running tests is easy -- a single directory
> contains tests for the entire framework, making it easy to offer a
> single autoloader for loading all library and test assets during
> testing. Because of the flexibility of PHPUnit, we can easily test a
> single component or class, or use annotations to group components and/or
> tests for running at once.
>
> Building the manual is also similarly easy: a single manual.xml file is
> used, and pulls in all manual pages via xincludes. Translation is fairly
> easy to accomplish, as we can copy the english sources into each
> translation directory, making it easy to run statistics on translation
> coverage. Fallbacks can be defined for xincludes to allow falling back
> to english if a translation is missing.
>
> If I understand correctly, you're positing that each component becomes a
> self-contained module, each containing code, tests, and documentation:
>
>      Zend-Form/
>          library/
>              Zend/
>                  Form/
>                      ....
>          tests/
>              Zend/
>                  Form/
>                      ...
>          documentation/
>              manual.xml
>              specs/
>                  ...
>
> This has huge ramifications in terms of how the repository is
> structured, how tests are run, and how the manual is built.
>
> On the plus side, the entire component is in one place, and it's easy to
> visualize what a package looks like for a discrete component. It would
> play very easily into any packaging system and distribution tool we
> might select.
>
> On the flip side:
>
>   * How does the component know where dependencies exist?
>      * This is important both when installed, as well as when *running*
>        *the* *tests*.
>   * How does documentation work?
>      * On the website, we'd want a single tree of manuals in order to
>        promote easy discovery. This implies a single build system -- how
>        would we accomplish that?
>      * Within a single module, do we build an individual manual for that
>        module for distribution? What about linking to other component
>        documentation -- how are those links resolved, particularly if
>        that component is not installed?
>
> Another point that comes to mind: you show individual adapters as being
> potentially modules -- why? Wouldn't that make things _more_ confusing?
> Shouldn't grabbing a component also grab all subcomponents such as
> adapters -- and why wouldn't we want these as a single package?
>
> I can _definitely_ see the pros to such an architecture -- don't get me
> wrong. I just also see a ton of HUGE infrastructure issues that would
> need to be addressed, and I think we have a lot of work to iron those
> out if we want to move forward. Frankly, I'm not sure anybody wants new
> development lingering for weeks or months before we figure those out,
> when the current infrastructure already is tried, tested, and proven.
>
> Feel free to try and convince me otherwise, though -- as I said, I can
> see the benefits; I'm just not sure they outweight the costs of change
> currently.
>
>> "Everything except for core is a module" will also meet the
>> requirement of openness and consistency. Because both zf-contributors
>> modules and "3rd party" modules are on thesame page, it is a truly
>> modular architecture. Before bashing this argument, please take a
>> moment to browse through Drupal and Wordpress. Because these are
>> CMSes, their core is not thin by ZF-standards, but it is ridiculously
>> thin by CMS standards. And, everything in these projects is a drop-in
>> module (also, peek again at the sheer number of modules available for
>> download for these projects)
>>
>> Consistent naming convention will greatly help with education.
>> Instead of saying:
>>      - grab the FormElements module, then download Captcha component
>>        and then look for ReCaptcha adapter which goes to xyz folder"
>>
>> we'll just say:
>>      - install ReCaptcha module
> Um, I think you've convoluted the story significantly, and not really
> detailed what you think you're trying to install here.
>
> If you're wanting to install the ReCaptcha form element, I don't see why
> it would be different than this:
>
>      prompt>  <installer>  install zf\Zend_Form
>      prompt>  <installer>  install zf\Zend_Service_ReCaptcha
>
> I think this would be true even in the modular architecture you envision
> -- the recaptcha form element consumes the recaptcha service as an
> optional dependency (if missing, it obviously fails all over the place,
> but you can still install the form component without it); the recaptcha
> service can be used without the form component.
>
> Can you provide a different example that better illustrates the
> differences in what has been proposed vs what you envision, please?
>
>> 3. THERE IS NO 3RD PARTY.  (also "Everything is miscellaneous").
>> To meet the requirement of openness, there can be no wall between us
>> (????), contributors and other people who just had an account set up
>> on github, or email at gmail. Contributing code is not the same as
>> writing stupid comments on facebook or uploading cat photos on
>> icanhazcheeseburger. It takes more knowledge and time, so there will
>> be less "spam". In terms of code quality, this is what GUIDELINES are
>> for.  We can have both! Openness and code quality guidelines.  We
>> _must not_ have a distinction between "us" and "them". There is no
>> "them".  We _must not_ have a wall of review process. The market does
>> that for us.  People will do that for us.
> I've outlined above why I think we need to make a differentiation
> between the official ZF distribution (the framework upon which
> applications are built) and a community forge/hub/etc. I'm not going
> over that again here.
>
>> I'm sorry, but trying to say that each and every ZF1 class is an
>> example of engineering brilliance is a huge overstatement ;)
> ZF1 has evolved tremendously over time. Many of the testing and
> documentation requirements also evolved -- most of these were missing
> prior to the 1.0 release, meaning that older code often is not
> engineered to the same standards that recent contributions are.
>
> Additionally, the PHP landscape has changed a ton in the last 5.5 years
> since the initial 0.1.0 release -- understanding and application of
> patterns, testing practices, etc. have taken a quantum leap forward
> since then. Yes, some code in ZF1 is better than others. The point is
> that we have standards that we've come to expect that we can now apply
> to the code base as a whole for ZF2.
>
>> How does it translate to real world? Well, some people just don't use
>> Zend_Db or Zend_Form, or Zend_Mail.  Same goes with drupal. Many
>> "standard modules" have been replaced by higher quality modules from
>> the catalog, from "the community". Now they are part of core or are
>> bundled in standard distribution.
> Having use-at-will installation answers your first question. Regarding
> the second part of your paragraph, that's what having a new major
> version allows us to do -- get rid of cruft and deprecated
> functionality, and add new code.
>
>> Modular architecture in this regard means, that each we give people a
>> choice of which and whose module I'll use for forms, and db and
>> mailing people.
> I see no conflict here. Assuming an installer tool that speaks to a
> server allowing developers to register their own repositories,
> developers can search for and install any component they want, be it
> from the official Zend repo or otherwise.
>
>> What about current zend-review components? Put them inside
>> repository/catalog ALONG with all other components. Because the CR
>> team has put a lot of effort and their time, we can have a
>> [zend-certified] badge or some other form of letting people know that
>> these components are _different_.  NOT "better"... "different".  You
>> can say "featured" for a lack of better word at this time.
>>
>> I'm sorry, but assuming that ZF review process somewhat makes ZF
>> components better than any external person in the world would write
>> them is just mathematically nearly impossible :-) I'm against
>> discrimination which will discourage any contributors that might just
>> find a better way to solve XYZ problem that the "almighty
>> zf-contributor ML subscribers".
> That's not the argument for inclusion in the ZF distribution AT ALL! The
> argument for inclusion is CAN WE and WILL WE SUPPORT and MAINTAIN the
> component long-term? I.e., is the component something that has a
> demonstrated need, and for which we will be able to find new maintainers
> should the original leave in the future?
>
> An official repo can easily live alongside a community repo -- or
> distributed individual repos. We can setup infrastructure that allows
> folks to search for functionality they desire from their installer, and
> which searches all registered repos. In that situation, the ZF repo is
> simply _one_ _source_ -- one that many modules/components/whatever might
> list dependencies against, but simply one source. This allows somebody
> to distribute their own form component, or DB implementation, or ORM, or
> whatever, and have it installed directly in their app or alongside the
> ZF installation (or set of ZF components they have installed).
>
>> If ZF2 became an open, modular, component framework it will create an
>> ecosystem.
> Yes, we need an ecosystem. I argue the way to do that is to be smart
> about selecting a packaging and installation tool that allows us to
> aggregate repos from disparate sources, including the ZF repo.
>
> The current processes are in place to create a story for maintenance and
> longevity of the code shipped directly with the framework. How
> individuals version and maintain their own repositories is up to them;
> we can simply make it easy for them to get their code in the hands of
> users.
>

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]



--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|

Re: Modular architecture != architecture for modules

Artur Bodera
In reply to this post by weierophinney
On Thu, Sep 1, 2011 at 5:58 PM, Matthew Weier O'Phinney <[hidden email]>wrote:

> >
> >  * flexibility
> >     Everything is optional. MVC is optional, Ralph's DB is optional,
> >     Form is optional, Tool, Autoloading and Application is optional (I
> >     know what I'm poking :) ).
>
> This is already what we're planning, though. If you want to use a single
> component, you grab it via the installer (whatever we end up choosing).
> Want just Zend\Db?
>
>    prompt> <installer> install zf\Zend_Db
>
> Does this differ from what you mean/intend? If so, how?


"I can feel disturbance in the force..."

More on that later.



>  >  * extendability and decoupling
> >     There is no single component, or there are very few that cannot be
> >     extended and/or replaced with a custom implementation. All
> >     components are equal in rights, ZF-reviewed or not.  All are
> >     obtainable and distributed the same way.
>
> Again, this is also a plan. One guiding principle for ZF2 has been that
> we try and follow SOLID principles when possible, and define interfaces
> for anything we think could provide an extension point -- which is a lot
> more often than not. Does this differ from what you mean/intend? If so,
> how?
>

Where does an "extension point" end and a "component" begin?


>
> Regarding all components being equal, "ZF-reviewed or not," I don't know
> that we can do that while we require a CLA, nor from a maintenance
> perspective. One big point of the proposal process is to identify
> components that we think will be of enough significant general use that
> we can commit to maintaining them long-term. Simply put, if there's not
> enough interest to maintain a component, we shouldn't be shipping it.
>

I'm against the "shipping" as a general concept.
CLA does not help with maintainability.
CLA does not help with people leaving (and definitely not coming in)
CLA does not help with people sitting on their butts and not fixing 2-yr old
bugs.
CLA does not help with code quality.

What does "general use" mean? Do we have any stats on that?

I'm sorry, but CLA is so old it trips over its own beard.
Who is processing CLA ? Zend Corp. from USA. Does CLA and ZF license take
international law into account? I'd say it's practically useless i.e. in
Poland.

CLA does not have to be banished completely, but it should morph to
something
"more 2011". For example - as per licensing - each Wordpress plugin (module)
has
to be GPL-compatible. Otherwise it will not be endorsed (read: published) on
the
wordpress.org website. From a developer point of view that gives similar
guarantees of code ownership as a ZF license + CLAs.




> That said, I _do_ think components should be obtainable and distributed
> in a consistent manner, whether or not they are an official part of the
> ZF distribution, or part of the larger ZF user/developer ecosystem.
> That's where finding a sane and powerful installation tool comes in;
> any tool we choose should allow for registering channels or repository
> sources in a central location to allow developers to search for
> components to use. Does this differ from what you intend? If so, how?
>

I am against what you call "ZF distribution". ZF2 should become a
microframework
with a huge ecosystem of downloadable components.

Please don't bring in the "local repositories" into the mix. I know most
pkg
managers allow devs to have localhost repositories with their own stuff etc.
That
is NOT what this discussion is about.

This discussion is about mitigating the trust gap, introducing openness and
eliminating all redundant review processes.



>  * Providing an installable package per component, plus
>  * A variety of meta-packages describing groupings of components
>
> answers this well. Based on the IRC discussion yesterday, you seemed to
> agree. Does this differ from what you intend? If so, how?
>
>
Only if its granular enough for full flexibility and allow drop-in comp.
replacement.



> It's not just about quality; it's about what is considered code that the
> ZF project _MAINTAINS_ and _SUPPORTS_. Having 3rd-party components share
> the same namespace makes it difficult for end-users to know who to go to
> for help, to report issues, or to submit patches.
>
> Openness is fine, and I fully support having a community hub for modules
> -- but we simply can't have folks using the "Zend" namespace for code
> we're not providing directly from the ZF project repository, for the
> reasons outlined in the previous paragraph.
>

This is the main point that divide us.

You are still referring to "ZF project" as an entity that lives, breaths
and
maintains and supports and can tie a tie.

On the other side of "the wall" [TM] you position "other people",
"community" or
whatever you'd like to call it.

First of all... it's 2011. It is NOT a problem to have a robust issue
tracker +VCS for
100 or 2000 contributors and 5000 different independent components.
Dev WILL know where to report issues and patches, because each module
is described with enough meta tags to be able to poke the person (people)
responsible. With VCS (and gems like github) you even know which person
to poke for a specific, _single line_ of buggy code.

Contributing to ZF is not a marriage, and even marriage is not constant.
You
can not predict nor stop ANYONE from leaving ZF, thus leaving component XYZ
unmaintained. You can not force anyone to fix outstanding, 2-year bugs.
Despite
these limitations (common to each and every open source project), Linux
project
is doing great! Magento too. Drupal is growing.


What you refer to "ZF project repository" should not exist... or should
allow anyone
to contribute code (i.e. ZF OPEN repository). The only requirement of
submission
should be agreeing to a common license terms and confirming that by
submitting
code, owner agrees to XYZ terms and confirms that he/she is the author (per
CLA).


Here are a few cases.

1) What if someone stops maintaining a module/component?

After some "buffer time", a new person can apply to take over the project.
If another person does not want to wait for it to come alive (see ZF1
components
that are unmaintained) then he/she can submit another version of the
component!
(i.e. Wildfire lost its maintainer, Wildfire2 pops up to replace it)


2) The tale of Zend\Db\Adapter\Firebird.

This is a perfect example of how a module = component. In this case, this
module
consists of 1 or a few php classes that implement Zend\Db\Adapter
interface.

Installing Zend\Db SHOULD NOT download all adapters! This is ridiculous in
several
ways. First of all, it should be granular. DB is the foundation, but I have
very, very
rarely seen a website use more than 1 different DB adapter at the same time.
On the other hand, there is maintainability. I don't know Matthew, how well
you know
Firebird SQL or the database specifics, but there are people who
use Firebird for living.
They know what they are doing and THEY should maintain an independent
Firebird
module.

Do not put it on Ralph shoulders.

Do not "bundle" (package?) it inside huge Zend\Db\* blob.

There are literally TENS of different
RDBMS<http://en.wikipedia.org/wiki/Comparison_of_relational_database_management_systems>
and
it's beyond one person ability to know
them all. Not mentioning doc storages, NoSQL (with new one popping up each
week).

Maintainability of Firebird CAN NOT affect the status of Mysql
adapter, or Zend\Db as
an API. It has to rest in the hands of people who will focus on Firebird
only.

The same principle applies to all "modules", as they can represent many
different
concepts I've described below.


3) The tale of Zend\Service retirement house.

This is a perfect example of why current review process, CLA, and project
organization
falls short in terms of allowing for developing service wrappers.
Programmableweb
lists 3797 <http://www.programmableweb.com/apis/directory>
public APIs for a myriad of different web services serving just any purpose
imaginable.
Are there more unmaintained Zend\Service\* classes in ZF1 than those
maintaned?
Each week I hear of a new "revolutionary" service which gives me goose
bumps. This
is the scenario where agile development is not an option, but a necessity.

According to my main concept and prime point, all Zend\Service classes
should become
modules in the "Services" group in module repository. Some of them will find
new maintainers,
some will just die off, but the openness will invite completely new
developers who have service
wrappers regardless of what we ramble about today.

How many of you (reading this now) have their own stash of classes,
components or whole
chunks of apps (ZF based or otherwise) that are maintained (because your
business demands
it), working and your are quite fond of it.

How many of you would submit them for other people to use, if there was not
cumbersome
review process, weeks of waiting and a number of formal requirements to
fulfill. How many
developers to date, who wanted to give away their solutions, have bounced
off ZF website after
trying to go through the convoluted spaghetti of regulations?




> The idea of a forge site for 3rd-party code has been raised a ton, and a
> thread or three have even been started on this ML amongst folks
> interested in building such a site. I'd love to see one; I also know
> that my team has too-limited resources to do it while also working on
> ZF2 and helping maintain ZF1. It has to be a community initiative at
> this point; however, once something workable is in place, we can
> certainly help support it in terms of providing a home (server and DNS)
> for it, as well as access for maintainers to assist.
>

Everything in ZF should be "3rd party".
Per your naming conventions, here is the proposed logical structure of ZF2:

ZF2 consists of
 * microframework CORE (highly optimized, tightly coupled, near-zero user
functionality)
 * modules/components organized in groups




> And I'd argue that the "gems" and some of the "popular, high-quality"
> components become candidates for inclusion in the main project itself.
>
> I don't see any conflict here, to be honest. With DVCS, it becomes
> trivial to merge such components into the master repository over time --
> while allowing the projects to maintain independence if desired, but
> also have independence while they work out a variety of kinks on a more
> agile basis before they suggest inclusion in the main project.
>
>
Again, there is the "master repository" terms that gives me the creeps.

ZF2 master repository should contain:
   * core (autoloading, tool, di, etc)
   * tests
   * examples

That is it. End. Complete. Fin!

The core should NEVER be extended with any real-world functionality.
This should all live inside "3rd party" components/modules.

Outside of ZF microframework core, there should be NO distinction
between Zend and non-Zend code.


> This is planned. Do you disagree with the core as defined in the
> distribution RFC? If so, what changes would you make?
>

It's actually very good. There will probably be little changes. I will
elaborate on that on dedicated
ML thread.


> I actually find the above _more_ confusing, to be honest. Why wouldn't
> all form-related classes be in a single module? and DB adapters? I could
> understand if they are _third-party_ offerings, but within the framework
> itself?
>

This is the definition of a microframework.
Microframework does not function on its own. For example, after going the
way I
described it, cloning ZF2 repository will result in having THE CORE. It will
not work
as an application, it will not have MVC, it will have absolutely no DB
connectivity.

After you "git clone zf2" you have to supply it (install) all components you
see fit
for your project at hand.



>
> Also, you've misquoted. The framework does not consist of "components,
> assets, meta-modules, packages, adapters, ...." That has never been
> stated. The framework as _currently_ proposed, consists of components --
> code, tests, and documentation only.
>
> _Distribution_ would allow selecting individual components and/or
> meta-packages consisting of multiple components. That's what the
> Distribution RFC was trying to identify.
>
> I think part of the problem I'm having is that I'm not seeing any
> concrete details on what a module, as you conceive it, _looks_ like, in
> the repository, nor how this impacts maintenance and packaging.
>

A "module" is a simplification of the above terms. It allows for a better
LOGICAL segmentation of the code.

From filesystem perspective, a module can be i.e.:
  * A single PHP class, i.e. Zend/Db/Adapter/Firebird.php
  * A directory of PHP classes, i.e.  Zend/Form/*
  * 20 PNG files
  * IndexController.php + routes.ini + pages.ini

From architectural perspective, a module can have one or many of these
functions:
  * provide PHP class(es) with a component (autoloaded and maintained by
"the core")
  * extend existing module (i.e. Db-ActiveRecord module on top of Db, or
Twig renderer module)
  * hook-up listeners and provide add-on functionality (i.e. debugging
console for MVC)
  * provide view script(s)
  * provide a collection of routes and/or routers

From functional perspective, a module can be:
  * a single adapter or provider
  * a single functional solution (aka component) that can be used by other
modules (i.e. Db)
  * a complete functional solution that depends on MVC module and provides
some form of UX.
  * a mutator, that changes how other module(s) behave (i.e. SSI caching
mechanism)
  * a transformator, that changes the output of other module(s) (i.e. YUI
output compressor)
  * a gateway, that allows or prevents other modules from working and/or
intercepts
    data flow (i.e. security module, anti-XSS module, auth module).



>
> In the current paradigm, running tests is easy -- a single directory
> contains tests for the entire framework, making it easy to offer a
> single autoloader for loading all library and test assets during
> testing. Because of the flexibility of PHPUnit, we can easily test a
> single component or class, or use annotations to group components and/or
> tests for running at once.
>

That doesn't change or affect what I propose.
A module will or will not contain tests. If a module consists of 20 PNG
files,
there will be no tests inside.



>
> Building the manual is also similarly easy: a single manual.xml file is
> used, and pulls in all manual pages via xincludes. Translation is fairly
> easy to accomplish, as we can copy the english sources into each
> translation directory, making it easy to run statistics on translation
> coverage. Fallbacks can be defined for xincludes to allow falling back
> to english if a translation is missing.
>
These are implementation details, while I'm trying to explain project
organisational and
high-level architectural drawbacks of ZF1.999


>
> If I understand correctly, you're positing that each component becomes a
> self-contained module, each containing code, tests, and documentation:
>
>    Zend-Form/
>        library/
>            Zend/
>                Form/
>                    ....
>        tests/
>            Zend/
>                Form/
>                    ...
>        documentation/
>            manual.xml
>            specs/
>                ...
>
> This has huge ramifications in terms of how the repository is
> structured, how tests are run, and how the manual is built.
>  On the plus side, the entire component is in one place, and it's easy to
> visualize what a package looks like for a discrete component. It would
> play very easily into any packaging system and distribution tool we
> might select.
>  On the flip side:
>
>  * How does the component know where dependencies exist?
>    * This is important both when installed, as well as when *running*
>      *the* *tests*.
>
Meta-data. It can be done.


>  * How does documentation work?
>    * On the website, we'd want a single tree of manuals in order to
>      promote easy discovery. This implies a single build system -- how
>      would we accomplish that?
>

No doc centralization. Each module should contain its own documentation.
If a module depends on another modul and "provides" for an API (interface)
then
we can import its docs upstream.

For example, browsing to the Db documentation would also query and show all
possible "Adapter modules" (providing or implementing Db\Adapter interface).
After
clicking Mysql we will be redirected to "Mysql module" documentation.



>    * Within a single module, do we build an individual manual for that
>      module for distribution? What about linking to other component
>      documentation -- how are those links resolved, particularly if
>      that component is not installed?
>

Linking down-the-tree implies that doc author is aware of existence of some
module X. This means, he/she know the main repo it lives in, so a link can
be constructed.

For example, if Db documentation mentions PostgreSQL as a "nice db to have"
it will link to non-existent "PostgreSQL module" documentation. Non-existing
docs will result in
   href="javascript:alert('Please first install module X to see its
documentation')"

It also means that module installation will invoke/propose docs
recompillation
for the installed module + upstream modules.

These are again implementation details. I will help with that in separate
thread that
popped up today on ML. Basically, we already somewhat accepted that code-
-scanning and file-sniffing will be necessary.

I believe that a momentary lost of performance (when scanning, because we
will have
a cache built later) does not outweights the benefits of having a
well-thought,
future-proof, granular and self-explanatory module infrastructure.

This has been mentioned by someone (you?) that most recent PHP projects
go in this direction, for reasons we can all relate to.



> Another point that comes to mind: you show individual adapters as being
> potentially modules -- why? Wouldn't that make things _more_ confusing?
> Shouldn't grabbing a component also grab all subcomponents such as
> adapters -- and why wouldn't we want these as a single package?
>

No.

Firebird module
jQuery module
AjaxForm module
ReCaptcha module

Db module
Db-with-all-adapters meta-package (linking to all @provides Db\Adapter)

Why not?
1. Maintainability
2. Flexibility
3. Separation of responsibility
4. Separation of concepts

It's repo-manager-engine's job to handle deps and expose convenience
meta-packages that will do what you described. This is probably more for
the education sake, as personally I (and probably many more experienced
devs) will prefer to get the microframework and then customize and fine-tune
it with selection of favorite modules.




> I can _definitely_ see the pros to such an architecture -- don't get me
> wrong. I just also see a ton of HUGE infrastructure issues that would
> need to be addressed, and I think we have a lot of work to iron those
> out if we want to move forward. Frankly, I'm not sure anybody wants new
> development lingering for weeks or months before we figure those out,
> when the current infrastructure already is tried, tested, and proven.
>
>
One important point here - I'm trying to promote more idealogical and
organizational changes. What I propose does not bring in additional work,
it is not heavy or difficult.

We do need a code scanner. We need lazy loading and meta-data processing
and installations and caching and config imports etc. What I propose will
not change that and we will have to hack it anyways.

My concepts will make the whole effort more worthwhile in the eyes of new
contributors and php developer ecosystem.



> Feel free to try and convince me otherwise, though -- as I said, I can
> see the benefits; I'm just not sure they outweight the costs of change
> currently.
>

Personally: please don't fiddle with the engineering details yet. The most
important
issue is organizational problems at this point. (see beginning of this
message).



>
> Can you provide a different example that better illustrates the
> differences in what has been proposed vs what you envision, please?
>

Above.


>
> ZF1 has evolved tremendously over time. Many of the testing and
> documentation requirements also evolved -- most of these were missing
> prior to the 1.0 release, meaning that older code often is not
> engineered to the same standards that recent contributions are.
>  Additionally, the PHP landscape has changed a ton in the last 5.5 years
> since the initial 0.1.0 release -- understanding and application of
> patterns, testing practices, etc. have taken a quantum leap forward
> since then. Yes, some code in ZF1 is better than others. The point is
> that we have standards that we've come to expect that we can now apply
> to the code base as a whole for ZF2.
>

Agreed. It's a fine piece of engineering. But it's like advertising a car by
saying
it's got the most advanced engine in the world, but got square wooden wheels
and tin-foil doors.

Current and recent departures from the projects and changes in the whole PHP
ecosystem are indication of things to come.

As I've outlined before, it is a HUGE challenge to retain high quality but
keep
the momentum going. Currently by average, even with recent events, ZF is
slowing down with each month. Not accelerating, slowing down and loosing its
shine.

It cannot go with the oligarchy model any more. As much as I value your work
folks, perceptional limitations of our brains are well documented and we can
NOT
push the whole train alone.



> I see no conflict here. Assuming an installer tool that speaks to a
> server allowing developers to register their own repositories,
> developers can search for and install any component they want, be it
> from the official Zend repo or otherwise.
>

Again with the official repo :-)

That's not the argument for inclusion in the ZF distribution AT ALL! The
> argument for inclusion is CAN WE and WILL WE SUPPORT and MAINTAIN the
> component long-term? I.e., is the component something that has a
> demonstrated need, and for which we will be able to find new maintainers
> should the original leave in the future?
>

Counter-argument at the beginning.
Everything changes.

No component and no person lives forever. I admire your strength in
defending
the castle, but ultimately its up to those individual folks that either
maintain their
little playgrounds, or not (with whatever excuse they have).




> An official repo can easily live alongside a community repo -- or
> distributed individual repos. We can setup infrastructure that allows
> folks to search for functionality they desire from their installer, and
> which searches all registered repos. In that situation, the ZF repo is
> simply _one_ _source_ -- one that many modules/components/whatever might
> list dependencies against, but simply one source. This allows somebody
> to distribute their own form component, or DB implementation, or ORM, or
> whatever, and have it installed directly in their app or alongside the
> ZF installation (or set of ZF components they have installed).
>
>
I cannot see to the future, but I will risk a prediction here. If there were
2 repositories:

 1. Zend Official CLA-guarded CR Team Reviewed and personally authographed
by MrGrumpy

 2. All-you-can-eat open repository for ZF


... I'd bet all my money and my cat on number 2.
As would Uncle Fester say: "Developers developers developers developers
...."
If 99% of components are in the repository no 2, then 99% of developers
(consuming) those
will come to repository no 2. This means even devels from no 1 will
double-think before
contributing to the official one, that is a zombie of a repo.

I hope you get my point. I'm tired.


A.

--
      __
     /.)\   +48 695 600 936
     \(./   [hidden email]
Reply | Threaded
Open this post in threaded view
|

Re: Modular architecture != architecture for modules

weierophinney
Administrator
-- Artur Bodera <[hidden email]> wrote
(on Thursday, 01 September 2011, 09:53 PM +0200):
> On Thu, Sep 1, 2011 at 5:58 PM, Matthew Weier O'Phinney <[hidden email]>wrote:

Replying to bits and pieces -- there's a ton to digest.

<snip>

> > In the current paradigm, running tests is easy -- a single directory
> > contains tests for the entire framework, making it easy to offer a
> > single autoloader for loading all library and test assets during
> > testing. Because of the flexibility of PHPUnit, we can easily test a
> > single component or class, or use annotations to group components and/or
> > tests for running at once.
>
> That doesn't change or affect what I propose.  A module will or will
> not contain tests. If a module consists of 20 PNG files, there will be
> no tests inside.

You missed the argument entirely. We run tests for the entire framework
at once right now, or individual components at a time.

My questions are: how does this change, if everything's a module?

For example, I want to run tests for a new Forms module. It
depends on functionality in Loader, Filter, Validator, and potentially
other components and/or modules. How do I inform the test harness of
their locations?

This may seem like an "implementation detail," but the fact is, the
current structure of the framework already solves this and allows us to
develop things _now_.

More below...

> > Building the manual is also similarly easy: a single manual.xml file is
> > used, and pulls in all manual pages via xincludes. Translation is fairly
> > easy to accomplish, as we can copy the english sources into each
> > translation directory, making it easy to run statistics on translation
> > coverage. Fallbacks can be defined for xincludes to allow falling back
> > to english if a translation is missing.
>
> These are implementation details, while I'm trying to explain project
> organisational and high-level architectural drawbacks of ZF1.999

Can you explain what these drawbacks are, exactly?

I _get_ that you're proposing a microframework architecture, but maybe
I'm having a failure of imagination here... what perceived problem does
it solve? what new problems does it enable us to solve?

>
> >
> > If I understand correctly, you're positing that each component becomes a
> > self-contained module, each containing code, tests, and documentation:
> >
> >    Zend-Form/
> >        library/
> >            Zend/
> >                Form/
> >                    ....
> >        tests/
> >            Zend/
> >                Form/
> >                    ...
> >        documentation/
> >            manual.xml
> >            specs/
> >                ...
> >
> > This has huge ramifications in terms of how the repository is
> > structured, how tests are run, and how the manual is built.
> >  On the plus side, the entire component is in one place, and it's easy to
> > visualize what a package looks like for a discrete component. It would
> > play very easily into any packaging system and distribution tool we
> > might select.
> >  On the flip side:
> >
> >  * How does the component know where dependencies exist?
> >    * This is important both when installed, as well as when *running*
> >      *the* *tests*.
> >
> Meta-data. It can be done.

I agree, it can be done -- but this is not an overnight task. The
current structure of the repository was created to make things like
testing, documentation, and translation simple. Need a new test? drop it
in the tests directory. Need to add some docs? drop something in the
documentation directory. Need to know what to translate? compare your
language's manual files against those in the english version.

A structure like the one above, to my mind, requires a ton of
infrastructure, assumptions, and potentially configuration to enable.
We, as a community, need to be very sure that the benefits outweigh the
cost of change at this point.

> >  * How does documentation work?
> >    * On the website, we'd want a single tree of manuals in order to
> >      promote easy discovery. This implies a single build system -- how
> >      would we accomplish that?
> >
>
> No doc centralization. Each module should contain its own
> documentation.  If a module depends on another modul and "provides"
> for an API (interface) then we can import its docs upstream.
>
> For example, browsing to the Db documentation would also query and
> show all possible "Adapter modules" (providing or implementing
> Db\Adapter interface).  After clicking Mysql we will be redirected to
> "Mysql module" documentation.

How do you propose to do this, exactly? What happens if somebody builds
just the "DB" docs -- how do we tell the doc compiler to build links
when documentation outside the module is referenced? This is a
non-trivial thing (I've tried before, actually).

I'm harping on things like docs and tests because they are part of a
contract with the users. A project is really only as good as its
documentation and its ability to prevent regressions (and add
features!). A very, very common complaint is the quality of
documentation -- and if we now have very fragmented documentation that
creates bad links or is incapable of referencing other portions of the
documentation, we've made the project harder for developers to pick up
and use.

> >    * Within a single module, do we build an individual manual for that
> >      module for distribution? What about linking to other component
> >      documentation -- how are those links resolved, particularly if
> >      that component is not installed?
> >
>
> Linking down-the-tree implies that doc author is aware of existence of
> some module X. This means, he/she know the main repo it lives in, so a
> link can be constructed.
>
> For example, if Db documentation mentions PostgreSQL as a "nice db to
> have" it will link to non-existent "PostgreSQL module" documentation.
> Non-existing docs will result in href="javascript:alert('Please first
> install module X to see its documentation')"
>
> It also means that module installation will invoke/propose docs
> recompillation for the installed module + upstream modules.

My point remains: the implementation details matter at this stage,
because if we adopt something like this, we now have the following:

 * Create new repository, or new empty branch of repository
 * Migrate code into repository, re-structuring how it fits
    * fixing unit tests
    * fixing docs
    * etc.

These tasks _will_ take time -- time that we _could_ be spending
developing new functionality or performing rewrites/refactoring.

<snip>

> > I can _definitely_ see the pros to such an architecture -- don't get me
> > wrong. I just also see a ton of HUGE infrastructure issues that would
> > need to be addressed, and I think we have a lot of work to iron those
> > out if we want to move forward. Frankly, I'm not sure anybody wants new
> > development lingering for weeks or months before we figure those out,
> > when the current infrastructure already is tried, tested, and proven.
> >
> One important point here - I'm trying to promote more idealogical and
> organizational changes. What I propose does not bring in additional work,
> it is not heavy or difficult.

But it does. We have code already existing, that is in a different
organizational structure. It *WILL* require work; don't claim it won't.
This is why we need to be very clear what the benefits would be, what
problems it solves, and what problems it allows us to solve.
Additionally, we need to know exactly how things would look, and
determine how much work it would take to accomplish such a change. Once
we know those things, we can weigh whether the benefits outweigh the
costs.

One last thing: I don't think it's an all-or-nothing decision. I think
the important aspect is a thriving community around the project; I don't
necessarily buy your assertion that the only way to do that is by your
suggestions. Successful projects exist that follow a large number of
organizational models.

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]