Quantcast

Config autoloading and merging -- revisited

classic Classic list List threaded Threaded
9 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Config autoloading and merging -- revisited

EvanDotPro
Hi all,

For a while now, the skeleton has contained a directory,
config/autoload/*. The idea behind this directory to allow a very
simple place for people to drop in their own configuration overrides
for modules, or even drop in "dist" config files provided by modules
which provide a template of configurable settings/options for a
module. I'd say this directory has quite successfully served its
purpose thus far.

I want to re-visit a discussion we had in a meeting a couple of months
back about configuration overrides and environment-specific configs.
Please keep in mind that what I'm discussing is simply about what we
provide as a suggestion in the skeleton application. **This does NOT
affect any framework code.**

So, historically when we first added the config/autoload directory, we
had it set up to load config/autoload/*.config.php.

In our meeting on 12-07-2011, due to questions about
environment-specific configs, we decided to introduce the concept of
"global" and "local" configs. To accomplish this, we changed the glob
pattern to what it is presently:
config/autoload/{,*.}{global,local}.config.php. See
http://framework.zend.com/wiki/display/ZFDEV2/2011-12-07+Meeting+Log

The idea is that anything ending in ".local.config.php" will be merged
after anything named *.global.config.php, meaning config from "local"
configs takes precedence over "global" configs, allowing you to
override config with local config files in each specific environment
you may have (development, testing, production). As a detail, the
beginning *. is optional, so you can also just have local.config.php
and global.config.php without prefixes. However, this change broke
some BC because no longer will a filename like foobar.config.php be
matched by the glob pattern. It must end in either .local.config.php
or .global.config.php. An example of something this broke is in
ZfcUser, I have a dist config file named
module.zfcuser.config.php.dist, which I had been instructing people to
copy (sans the .dist) to their autoload directory and edit to override
configuration. Now we'd have to rename that to something like
module.zfcuser.global.config.php.dist. Not only is this getting pretty
long, I'm not really fond of pre-determining that the user wants the
settings in that file to be global, and it could cause some confusion
for beginners about what can/can't/should/shouldn't be overridden for
a module.

So with that explained, here's some thoughts and questions I had:

- I'd like to remove the ".config" part of the pattern as a
requirement, and make it optional. With these files being under a
config/ directory, I find having "config" in the filename a bit
redundant. You'll also notice in the meeting log linked above, there
was no mention of having config in the filenames. The fact that it
remains in the glob pattern is simply a leftover artifact of the
earlier implementation.

- Do we want to *require* the "local" and "global" parts in the
filename with the default glob pattern in the skeleton, or should
those just be optional? If they're optional, that would create three
"groups" of config files: global, local, and neither... The reason for
not simply doing *.php is that based on the current recommendation, we
want "local" configs to be merged before and take precedence over
global config files. By specifying the glob pattern in braces for the
two, we can dictate the order the files are returned by glob() in.

If we do want to require local or global, we could use a glob like
"config/autoload/{,*.}{global,local}.php" however it's not quite
right:

    [0] => config/autoload/global.php
    [1] => config/autoload/local.php
    [2] => config/autoload/00-database.global.php
    [3] => config/autoload/bbb.global.php
    [4] => config/autoload/00-database.local.php
    [5] => config/autoload/aaa.local.php

Something like "config/autoload/{global,*.global,local,*.local}.php"
gets us a little closer to what I'd expect:

    [0] => config/autoload/global.php
    [1] => config/autoload/00-database.global.php
    [2] => config/autoload/bbb.global.php
    [3] => config/autoload/local.php
    [4] => config/autoload/00-database.local.php
    [5] => config/autoload/aaa.local.php

(The local..php is intentional just to help illustrate how the glob is
matching.)

Making the global/local parts optional could prove a little more
tricky. We could do it with multiple globs or a complex glob but
there'd likely be overlap in the matches (one file matched multiple
times), so we might want to look into running array_unique(glob()) or
something along those lines.

- Should we promote (not enforce in any way) the use of numeric
prefixes for autoloaded config files? In the *nix world, this is a
common practice in conf.d/* situations where in /etc/X11/xorg.conf.d/,
you might find something like 00-system-setup-keyboard.conf
10-synaptics.conf. My argument for this is that it's very simple to
explain and document this concept for new users. This practice can be
used in conjunction with the global/local concept to allow for more
fine-grained control within groups.

Thanks,

--
Evan Coury
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Config autoloading and merging -- revisited

weierophinney
Administrator
-- Evan Coury <[hidden email]> wrote
(on Saturday, 31 March 2012, 05:31 AM -0700):
<snip>
> So with that explained, here's some thoughts and questions I had:
>
> - I'd like to remove the ".config" part of the pattern as a
> requirement, and make it optional.

This makes sense to me.

> - Do we want to *require* the "local" and "global" parts in the
> filename with the default glob pattern in the skeleton, or should
> those just be optional?

Optional. Some configurations do not require "local" variants (for
instance, there may not be any settings that need to be overridden
per-environmrnet).

> - Should we promote (not enforce in any way) the use of numeric
> prefixes for autoloaded config files? In the *nix world, this is a
> common practice in conf.d/* situations where in /etc/X11/xorg.conf.d/,
> you might find something like 00-system-setup-keyboard.conf
> 10-synaptics.conf. My argument for this is that it's very simple to
> explain and document this concept for new users. This practice can be
> used in conjunction with the global/local concept to allow for more
> fine-grained control within groups.

I'd say this should be documented as a way to provide precedence _when_
_needed_. This is something that will vary from project to project,
really.

I have some more suggestions to make. :)

- Expand the glob pattern to allow .php, .xml, .ini, and whatever other
  config formats we allow, and make sure the Config listener uses the
  factory in order to load configuration.

- GLOB_BRACE only works on some architectures. I realize that Ben has
  created a PR that basically emulates GLOB_BRACE in userland, but I'm
  wondering if perhaps instead we should recommend adding several glob
  patterns in succession? This would simplify the logic internally (in
  fact, I think you already implemented this), and keep us platform
  agnostic.

- I saw a PR recently that suggested adding the ability to provide
  _static_ paths to the Config merger -- in other words, no globs, just
  the path. This gives us the "ultimate" in both speed and specifying
  order, which would be good for production; in development, clearly, it
  would stink. :)

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Config autoloading and merging -- revisited

akrabat
In reply to this post by EvanDotPro

On 31 Mar 2012, at 13:31, Evan Coury wrote:

> So with that explained, here's some thoughts and questions I had:
>
> - I'd like to remove the ".config" part of the pattern as a
> requirement, and make it optional. With these files being under a
> config/ directory, I find having "config" in the filename a bit
> redundant. You'll also notice in the meeting log linked above, there
> was no mention of having config in the filenames. The fact that it
> remains in the glob pattern is simply a leftover artifact of the
> earlier implementation.

Fine with me.


> - Do we want to *require* the "local" and "global" parts in the
> filename with the default glob pattern in the skeleton, or should
> those just be optional? If they're optional, that would create three
> "groups" of config files: global, local, and neither... The reason for
> not simply doing *.php is that based on the current recommendation, we
> want "local" configs to be merged before and take precedence over
> global config files. By specifying the glob pattern in braces for the
> two, we can dictate the order the files are returned by glob() in.

I think it's clearer with the "section name" within the filename.

>
> If we do want to require local or global, we could use a glob like
> "config/autoload/{,*.}{global,local}.php" however it's not quite
> right:
>
>    [0] => config/autoload/global.php
>    [1] => config/autoload/local.php
>    [2] => config/autoload/00-database.global.php
>    [3] => config/autoload/bbb.global.php
>    [4] => config/autoload/00-database.local.php
>    [5] => config/autoload/aaa.local.php
>
> Something like "config/autoload/{global,*.global,local,*.local}.php"
> gets us a little closer to what I'd expect:
>
>    [0] => config/autoload/global.php
>    [1] => config/autoload/00-database.global.php
>    [2] => config/autoload/bbb.global.php
>    [3] => config/autoload/local.php
>    [4] => config/autoload/00-database.local.php
>    [5] => config/autoload/aaa.local.php
>
> (The local..php is intentional just to help illustrate how the glob is
> matching.)

Certainly, I'd expect any filename containing "local" to be loaded after any filename containing "global".

Ideally, I'd like to see module.XXX.php first, then global then local. I'd almost argue for putting the section name first so that the files for a given section are grouped in the filesystem, but I'm not fussy on this.

This would result in something like:

   [0] => config/autoload/module.zfcuser.php
   [1] => config/autoload/module.doctrine.php
   [2] => config/autoload/global.php
   [3] => config/autoload/global.00-database.php
   [4] => config/autoload/global.bbb.php
   [5] => config/autoload/local.php
   [6] => config/autoload/local.00-database.php
   [7] => config/autoload/local.aaa.php

Personally, I would also like to see out of the box support for APPLICATION_ENV as this is already an FAQ and a remarkably common requirement.



> - Should we promote (not enforce in any way) the use of numeric
> prefixes for autoloaded config files? In the *nix world, this is a
> common practice in conf.d/* situations where in /etc/X11/xorg.conf.d/,
> you might find something like 00-system-setup-keyboard.conf
> 10-synaptics.conf. My argument for this is that it's very simple to
> explain and document this concept for new users. This practice can be
> used in conjunction with the global/local concept to allow for more
> fine-grained control within groups.

My personal preference is that you shouldn't have to care about order within module/global/local. For the few people that find they actually do need to care about order, then we can have a KB article explaining how to do it.


Regards,

Rob…

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Config autoloading and merging -- revisited

Mike Willbanks
Hello

<snip>

>> Something like "config/autoload/{global,*.global,local,*.local}.php"
>> gets us a little closer to what I'd expect:
>>
>>    [0] => config/autoload/global.php
>>    [1] => config/autoload/00-database.global.php
>>    [2] => config/autoload/bbb.global.php
>>    [3] => config/autoload/local.php
>>    [4] => config/autoload/00-database.local.php
>>    [5] => config/autoload/aaa.local.php
>>
>> (The local..php is intentional just to help illustrate how the glob is
>> matching.)
>
> Certainly, I'd expect any filename containing "local" to be loaded after any filename containing "global".
>
> Ideally, I'd like to see module.XXX.php first, then global then local. I'd almost argue for putting the section name first so that the files for a given section are grouped in the filesystem, but I'm not fussy on this.
>
> This would result in something like:
>
>   [0] => config/autoload/module.zfcuser.php
>   [1] => config/autoload/module.doctrine.php
>   [2] => config/autoload/global.php
>   [3] => config/autoload/global.00-database.php
>   [4] => config/autoload/global.bbb.php
>   [5] => config/autoload/local.php
>   [6] => config/autoload/local.00-database.php
>   [7] => config/autoload/local.aaa.php
>
> Personally, I would also like to see out of the box support for APPLICATION_ENV as this is already an FAQ and a remarkably common requirement.

I completely agree here; this is something that I am currently doing
manually.  We actually do this separately today (but could just add
the environment to the glob so to speak.  We actually have
environmental configurations vs. local configurations.  This is on
purpose so that they can actually be version controlled.  For
instance, when I make an update, I can go through all of my developers
configuration files and ensure that they are going to be compatible
with the changes that I am going to push.  This minimizes the amount
of communication needed as well as any effort of work and saves a ton
of questions.

Additionally, since the configuration files are kept in version
control; it aides us in assisting the system administrators in keeping
the configuration up to date across the cluster of servers.

<snip>

Regards,

Mike
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Config autoloading and merging -- revisited

SocalNick
Personally I like the removal of APPLICATION_ENV. We are planning on handling version control of our local.config.php by putting it in the hands of our TechOps team. They use Chef / OpsCode to deploy our server farms and we feel this is where local configuration belongs. Configuration for our software stack is in their hands, we just think of our ZF2 app as another piece of the software stack that they install and configure. Another consideration for security: everyone with access to the Git repo should not necessarily know the production database passwords.

-Nick


On Tue, Apr 3, 2012 at 7:19 AM, Mike Willbanks <[hidden email]> wrote:
Hello

<snip>

>> Something like "config/autoload/{global,*.global,local,*.local}.php"
>> gets us a little closer to what I'd expect:
>>
>>    [0] => config/autoload/global.php
>>    [1] => config/autoload/00-database.global.php
>>    [2] => config/autoload/bbb.global.php
>>    [3] => config/autoload/local.php
>>    [4] => config/autoload/00-database.local.php
>>    [5] => config/autoload/aaa.local.php
>>
>> (The local..php is intentional just to help illustrate how the glob is
>> matching.)
>
> Certainly, I'd expect any filename containing "local" to be loaded after any filename containing "global".
>
> Ideally, I'd like to see module.XXX.php first, then global then local. I'd almost argue for putting the section name first so that the files for a given section are grouped in the filesystem, but I'm not fussy on this.
>
> This would result in something like:
>
>   [0] => config/autoload/module.zfcuser.php
>   [1] => config/autoload/module.doctrine.php
>   [2] => config/autoload/global.php
>   [3] => config/autoload/global.00-database.php
>   [4] => config/autoload/global.bbb.php
>   [5] => config/autoload/local.php
>   [6] => config/autoload/local.00-database.php
>   [7] => config/autoload/local.aaa.php
>
> Personally, I would also like to see out of the box support for APPLICATION_ENV as this is already an FAQ and a remarkably common requirement.

I completely agree here; this is something that I am currently doing
manually.  We actually do this separately today (but could just add
the environment to the glob so to speak.  We actually have
environmental configurations vs. local configurations.  This is on
purpose so that they can actually be version controlled.  For
instance, when I make an update, I can go through all of my developers
configuration files and ensure that they are going to be compatible
with the changes that I am going to push.  This minimizes the amount
of communication needed as well as any effort of work and saves a ton
of questions.

Additionally, since the configuration files are kept in version
control; it aides us in assisting the system administrators in keeping
the configuration up to date across the cluster of servers.

<snip>

Regards,

Mike

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Config autoloading and merging -- revisited

EvanDotPro
On Tue, Apr 3, 2012 at 9:52 AM, Nicholas Calugar <[hidden email]> wrote:
> Personally I like the removal of APPLICATION_ENV. We are planning on
> handling version control of our local.config.php by putting it in the hands
> of our TechOps team. They use Chef / OpsCode to deploy our server farms and
> we feel this is where local configuration belongs. Configuration for our
> software stack is in their hands, we just think of our ZF2 app as another
> piece of the software stack that they install and configure. Another
> consideration for security: everyone with access to the Git repo should not
> necessarily know the production database passwords.

This has been my train of thought as well; however it's interesting to
see how divided people are on this topic. The nice thing about how
it's being handled now is that it's just one very small and simple
change to add ENV config file support to what we already have with
global/local. Saying "we have to do this manually now" seems a bit
silly considering it's simply making one small edit to one line in
index.php. The plan has been to default to the local/global and
document how to do named ENV specific configs for people who feel they
still need them.

--
Evan Coury
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Config autoloading and merging -- revisited

akrabat
In reply to this post by SocalNick

On 3 Apr 2012, at 17:52, Nicholas Calugar wrote:

> Personally I like the removal of APPLICATION_ENV. We are planning on handling version control of our local.config.php by putting it in the hands of our TechOps team. They use Chef / OpsCode to deploy our server farms and we feel this is where local configuration belongs. Configuration for our software stack is in their hands, we just think of our ZF2 app as another piece of the software stack that they install and configure. Another consideration for security: everyone with access to the Git repo should not necessarily know the production database passwords.
>
> -Nick

That's fine for companies with TechOps teams with server farms :)  
I agree re passwords, which is why I quite like local.php too. For us though, it's not just passwords that have different config settings dev/staging/production; we also have config for email, remote web service URLs, logging etc which we currently put in our VCS. Obviously, we're in a minority.

As I said though, its already becoming a FAQ, so the documentation to make this change has to be very easy to find.  


Regards,

Rob...
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Config autoloading and merging -- revisited

Sascha-Oliver Prolic
In reply to this post by SocalNick
Am 3. April 2012 18:52 schrieb Nicholas Calugar <[hidden email]>:

> Personally I like the removal of APPLICATION_ENV. We are planning on
> handling version control of our local.config.php by putting it in the hands
> of our TechOps team. They use Chef / OpsCode to deploy our server farms and
> we feel this is where local configuration belongs. Configuration for our
> software stack is in their hands, we just think of our ZF2 app as another
> piece of the software stack that they install and configure. Another
> consideration for security: everyone with access to the Git repo should not
> necessarily know the production database passwords.
>
> -Nick
>
Hi Nick, hi all,

for situations like that, i take another repository, unknown to the
developers, the build server (jenkins, bamboo, whatever) has access to
the special repository and pulls out the configs with the passwords
for database access etc.

About using APPLICATION_ENV:
Let me explain this from a ZF1 point of view. I integrated a homegrown
"Service Locator Generator" (yes, something with a compatible
interface to Zend\Di\Locator) into ZF1 and completely removed
Zend\Application therefore. (Code is not public currently, because ZF2
is coming out soon, but if someone is interessted in it, i'll put it
on github.) So what about APPLICATION_ENV in my ZF1-app? It is
completely removed!! The only thing that i have is the following:

application/configs/di/zend.php
application/configs/di/doctrine.php
application/configs/di/application.php
.....
application/configs/di/settings.production.php
application/configs/di/settings.development.php
application/configs/di/settings.staging.php

As I said, it is only a service locator generator, so every time when
i edit one of these di config files, i need to go in my shell and
type:
% php sl_generator.php -e development
-e is the environment switch, I'll tell my service locator generator
which environment to choose, therefore my code is completely unknown
of any application environment.

Back to ZF2:
I would expect having no APPLICATION_ENV at all in ZF2. But the DI
implementation can be used dynamically (@see RuntimeDefinition), so
how can we enforce the same thing? Well first option is reinvent
APPLICATION_ENV to enforce which config to load. Other option is how i
tried to explain above:
copy a local.config.php.dist to the config dir and rename it to
local.config.php, and use a seperate repository to hide the database
passwords from developers for production use. allow perhabs to have a
second dist-config file with special settings (could think of
replacing activeMQ with a DB-Based Queue on developer machines but use
activeMQ on staging and production, diffent ldap connection points,
etc.).

Regards

Sascha-Oliver Prolic
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Config autoloading and merging -- revisited

mtymek
This post has NOT been accepted by the mailing list yet.
> Code is not public currently, because ZF2
> is coming out soon, but if someone is interessted in it, i'll put it
> on github.

Please post it, I'm interested! :-)
Loading...