Quantcast

Glob path config merging

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Glob path config merging

EvanDotPro
Hi all,

Recently, due to several requests from the community, I added the
ability for developers to define a glob path to merge configuration
outside of the scope of modules.

For example, in index.php, before calling
$moduleManager->loadModules(); you can do this (as seen in the latest
ZendSkeletonApplication):

$moduleManager->getConfigListener()->addConfigGlobPath(dirname(__DIR__)
. '/config/autoload/*.config.php');

The relevant code is here:
https://github.com/zendframework/zf2/blob/master/library/Zend/Module/Listener/ConfigListener.php

However, this poses a problem: how to handle the environment string in
these autoloaded config files. In modules, this problem is
non-existent because all the framework cares is that
FooModule\Module::getConfig($env) returns the config. How or if it
dealt with $env is of no concern, it's up to the module. However, now
that we've added this glob() config loading mechanism, the picture is
not so clear.

One useful purpose for this feature is that modules can provide nice,
clean dist config files that developers can simply copy into their
./config/autoload directory and be on their way (thus discouraging
editing config files from within modules themselves which is a no-no).
However, if we always try to load the section of the config file
matching the environment, then that forces module developers to push
something like this in their dist configs (php config as an example
here):

return array(
    'development' => $config,
    'testing' => $config,
    'staging' => $config,
    'production' => $config,
);

And even then, it's not guaranteed to work. Those are just sample
environment strings, there's an infinite number of possibilities.
Simply put: module dist configs shouldn't have to guess what the env
string is for an app. That is a poor solution, and many developers
would have to go in and further customize what should have been a
drop-in config file to match their used environment strings. Not to
mention, there will be many modules that have basic configs that are
not inherently sensitive to the environment.

So on IRC, two main ideas have been thrown around:

1) Do not utilize the concept of config sections at all, and use a
more dynamic glob() to grab both ./config/autoload/*.config.php AND
./config/autoload/environment/*.config.$env.php. Obviously the
disadvantage is that you can't use config sections, and MUST separate
the environments into different files (some people prefer this, others
don't).
2) Allow sections to be used, but if the given section doesn't exist,
just use the full config array provided by the file instead of
throwing an exception like it currently does. This has the obvious
disadvantage of the system not loading the config properly if a config
file is meant to be sectioned based on env, but the env in question is
not included in the config file. It will silently use the config file
in the wrong manner.

So here's a hybrid of the two, both of which have the disadvantages
stated above:

First, loop through config files matching a semi-dynamic glob()
something like this:

$moduleManager->getConfigListener()->addConfigGlobPath(
    dirname(__DIR__)
    . '/etc/conf.d/{*.config.php,environment/*.config.'
    . $listenerOptions->getApplicationEnvironment()
    . '.php}'
);

Next, if (isset($config[$env])) { /* use the env key of the config */
} else { /* use the full config */ }

With this, when the app env is set to 'development', it will load all
./config/autoload/environment/*.config.development.php files. They can
either be sectioned or not, it doesn't matter. Same with the config
files in ./config/autoload/*.config.php.

Personally, I'm not excited about either solution. This type of
problem is a perfect example of why for modules, we have delegating
the responsibility of handling (or not) the environment parameter to
the module class' getConfig() method. The reason we had originally
suggested that configs are overridden via a last-loaded module instead
of this glob thing is that then each developer gets to choose how they
load configs and handle the $env parameter in their getConfig()
method.

The problem we have now that some people prefer to have the framework
also able to directly inject config without having a module class that
returns it. As a result, we now have this glob thing that is
framework-managed instead of module-managed and we need to figure out
how it should work.

So basically, who has some better ideas?

---
Evan Coury
http://blog.evan.pro/

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glob path config merging

keith Pope-4
On 28 November 2011 20:02, Evan Coury <[hidden email]> wrote:

> Hi all,
>
> Recently, due to several requests from the community, I added the
> ability for developers to define a glob path to merge configuration
> outside of the scope of modules.
>
> For example, in index.php, before calling
> $moduleManager->loadModules(); you can do this (as seen in the latest
> ZendSkeletonApplication):
>
> $moduleManager->getConfigListener()->addConfigGlobPath(dirname(__DIR__)
> . '/config/autoload/*.config.php');
>
> The relevant code is here:
> https://github.com/zendframework/zf2/blob/master/library/Zend/Module/Listener/ConfigListener.php
>
> However, this poses a problem: how to handle the environment string in
> these autoloaded config files. In modules, this problem is
> non-existent because all the framework cares is that
> FooModule\Module::getConfig($env) returns the config. How or if it
> dealt with $env is of no concern, it's up to the module. However, now
> that we've added this glob() config loading mechanism, the picture is
> not so clear.
>
> One useful purpose for this feature is that modules can provide nice,
> clean dist config files that developers can simply copy into their
> ./config/autoload directory and be on their way (thus discouraging
> editing config files from within modules themselves which is a no-no).
> However, if we always try to load the section of the config file
> matching the environment, then that forces module developers to push
> something like this in their dist configs (php config as an example
> here):
>
> return array(
>    'development' => $config,
>    'testing' => $config,
>    'staging' => $config,
>    'production' => $config,
> );
>
> And even then, it's not guaranteed to work. Those are just sample
> environment strings, there's an infinite number of possibilities.
> Simply put: module dist configs shouldn't have to guess what the env
> string is for an app. That is a poor solution, and many developers
> would have to go in and further customize what should have been a
> drop-in config file to match their used environment strings. Not to
> mention, there will be many modules that have basic configs that are
> not inherently sensitive to the environment.
>
> So on IRC, two main ideas have been thrown around:
>
> 1) Do not utilize the concept of config sections at all, and use a
> more dynamic glob() to grab both ./config/autoload/*.config.php AND
> ./config/autoload/environment/*.config.$env.php. Obviously the
> disadvantage is that you can't use config sections, and MUST separate
> the environments into different files (some people prefer this, others
> don't).
> 2) Allow sections to be used, but if the given section doesn't exist,
> just use the full config array provided by the file instead of
> throwing an exception like it currently does. This has the obvious
> disadvantage of the system not loading the config properly if a config
> file is meant to be sectioned based on env, but the env in question is
> not included in the config file. It will silently use the config file
> in the wrong manner.
>
> So here's a hybrid of the two, both of which have the disadvantages
> stated above:
>
> First, loop through config files matching a semi-dynamic glob()
> something like this:
>
> $moduleManager->getConfigListener()->addConfigGlobPath(
>    dirname(__DIR__)
>    . '/etc/conf.d/{*.config.php,environment/*.config.'
>    . $listenerOptions->getApplicationEnvironment()
>    . '.php}'
> );
>
> Next, if (isset($config[$env])) { /* use the env key of the config */
> } else { /* use the full config */ }
>
> With this, when the app env is set to 'development', it will load all
> ./config/autoload/environment/*.config.development.php files. They can
> either be sectioned or not, it doesn't matter. Same with the config
> files in ./config/autoload/*.config.php.
>
> Personally, I'm not excited about either solution. This type of
> problem is a perfect example of why for modules, we have delegating
> the responsibility of handling (or not) the environment parameter to
> the module class' getConfig() method. The reason we had originally
> suggested that configs are overridden via a last-loaded module instead
> of this glob thing is that then each developer gets to choose how they
> load configs and handle the $env parameter in their getConfig()
> method.
>
> The problem we have now that some people prefer to have the framework
> also able to directly inject config without having a module class that
> returns it. As a result, we now have this glob thing that is
> framework-managed instead of module-managed and we need to figure out
> how it should work.
>
> So basically, who has some better ideas?

Agreed both situations sound pretty nasty to me, maybe its a bad idea????

The only other solution I could think of would be to have a available
environments config somewhere, so you know the config sections to look
for, I suppose then you would end up having to map sections to other
sections if you have two modules with different env names that are
equal. Again a bad situation to be in.... What if you have a
environment mapper that could be compiled for production using the
cli? This way in dev we could take the pointless overhead and in
production merge everything together.

Maybe if you use the glob path style you have to use env directories,
else you config them within the module?

thats my 2cents anyway :)

>
> ---
> Evan Coury
> http://blog.evan.pro/
>
> --
> List: [hidden email]
> Info: http://framework.zend.com/archives
> Unsubscribe: [hidden email]
>
>



--
------------
http://www.thepopeisdead.com

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glob path config merging

EvanDotPro
On Tue, Nov 29, 2011 at 7:49 AM, keith Pope <[hidden email]> wrote:

> Agreed both situations sound pretty nasty to me, maybe its a bad idea????
>
> The only other solution I could think of would be to have a available
> environments config somewhere, so you know the config sections to look
> for, I suppose then you would end up having to map sections to other
> sections if you have two modules with different env names that are
> equal. Again a bad situation to be in.... What if you have a
> environment mapper that could be compiled for production using the
> cli? This way in dev we could take the pointless overhead and in
> production merge everything together.

Well, we do already have config caching in the config listener. This
takes into account the environment and generates a cache file per
environment. Once the cache file is generated (happens at runtime
after config cache is enabled), then subsequent requests completely
skip all calls to module getConfig() methods, glob merging, etc. It
just simply includes the single cached array for the given
environment.

--
Evan Coury

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glob path config merging

guilhermeblanco@gmail.com
I pretty much discussed this with Evan on IRC, but here is my opinion.

I always hated the idea of having sections defining the environment,
specially because they tend to expose production settings to all
developers, while it should be visible to buildmaster only.

I'm a huge fan of localized files, and we could use the sections as
component based configuration.
Finally, we could merge global and local files according to purposed
(tests, normal).

My wish is that we could standardize something like this:

$moduleManager->getConfigListener()->addConfigGlobPath(
   dirname(__DIR__) . '/etc/conf.d/*.{global,local}.php'
);

That way we can have:

route.global.php
route.local.php

When running the test environment, I'd just choose {global,test} and
we'd be done.
Since it'd be a .gitignore'd file, I would also recommend to
distribute a route.local.php.dist (or any other resource that you
want, even application.local.php.dist).


Cheers,

On Tue, Nov 29, 2011 at 10:06 AM, Evan Coury <[hidden email]> wrote:

> On Tue, Nov 29, 2011 at 7:49 AM, keith Pope <[hidden email]> wrote:
>> Agreed both situations sound pretty nasty to me, maybe its a bad idea????
>>
>> The only other solution I could think of would be to have a available
>> environments config somewhere, so you know the config sections to look
>> for, I suppose then you would end up having to map sections to other
>> sections if you have two modules with different env names that are
>> equal. Again a bad situation to be in.... What if you have a
>> environment mapper that could be compiled for production using the
>> cli? This way in dev we could take the pointless overhead and in
>> production merge everything together.
>
> Well, we do already have config caching in the config listener. This
> takes into account the environment and generates a cache file per
> environment. Once the cache file is generated (happens at runtime
> after config cache is enabled), then subsequent requests completely
> skip all calls to module getConfig() methods, glob merging, etc. It
> just simply includes the single cached array for the given
> environment.
>
> --
> Evan Coury
>
> --
> List: [hidden email]
> Info: http://framework.zend.com/archives
> Unsubscribe: [hidden email]
>
>



--
Guilherme Blanco
MSN: [hidden email]
GTalk: guilhermeblanco
Toronto - ON/Canada

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glob path config merging

EvanDotPro
On Mon, Dec 5, 2011 at 9:38 PM, [hidden email]
<[hidden email]> wrote:

> I pretty much discussed this with Evan on IRC, but here is my opinion.
>
> I always hated the idea of having sections defining the environment,
> specially because they tend to expose production settings to all
> developers, while it should be visible to buildmaster only.
>
> I'm a huge fan of localized files, and we could use the sections as
> component based configuration.
> Finally, we could merge global and local files according to purposed
> (tests, normal).
>
> My wish is that we could standardize something like this:
>
> $moduleManager->getConfigListener()->addConfigGlobPath(
>   dirname(__DIR__) . '/etc/conf.d/*.{global,local}.php'
> );
>
> That way we can have:
>
> route.global.php
> route.local.php
>
> When running the test environment, I'd just choose {global,test} and
> we'd be done.
> Since it'd be a .gitignore'd file, I would also recommend to
> distribute a route.local.php.dist (or any other resource that you
> want, even application.local.php.dist).

I'll add that this is also my preferred method of handling
environment-specific configs, NOT using config sections. In fact, I've
been using environment-local configs that are excluded from source
control for several years successfully without ever relying on
environment configs as recommended in ZF1 with Zend\Application. I
know there are some with different preferences, I'm just saying it has
worked great for me in all sorts of projects (small personal, large
enterprise, etc).

As far as dividing the configs up for things like routes, db
credentials, etc. That's definitely already possible, and in fact I'd
encourage such a practice for the sake of configuration organization
and management.

--
Evan Coury

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glob path config merging

Wil Moore III
This post has NOT been accepted by the mailing list yet.
I _mostly_ agree with this sentiment; however, I reply with some personal opinions below:

> I always hated the idea of having sections defining the environment, 

While I wouldn't say I love them, I can't say I hate them either. It is a mixed bag. A section per environment has the following pros:

1 - Easy to extend and override an environment

Cons:

1 - Lots of duplication (some of this is mitigated depending on the config format you choose -- for instance, this is less of an issue w/ Yaml, but still an issue)

2 - Need to add a new configuration section each time you add a uniquely named environment

3 - A large file with many differences per environment and many environments can start to get pretty unwieldy. I make no performance claims (as I haven't noticed anything significant, nor have I benchmarked for this) with this.

> specially because they tend to expose production settings to all developers, while it should be visible to buildmaster only. 

I would challenge this assertion as I believe this is only an issue if you aren't doing token replacement on your config files at build time. That being said, even if you are doing this, the down-side with a section per environment is that _ALL_ sections are replaced. The configuration will still work as expected if you ignore this detail; however, this is just plain confusing and annoying.

But, the main point I make here is that if you use .properties files and replace tokens in the config file (ant, gradle, phing, etc.) then most of this goes away.

Doing so completely mitigates the issue of production information being in the hands of those that don't need to know. If you are doing CI (Jenkins, etc.) then this is even easier as you're build server holds on to these .properties files and your build script just pulls them in and uses them.

> I'm a huge fan of localized files, and we could use the sections as component based configuration. 
> Finally, we could merge global and local files according to purposed 
> (tests, normal). 

This seems similar to what I mentioned above (token replacement); however, I feel like token replacement is fairly ubiquitous (in the...is there a build community?) :).


So, from my perspective, sections per environment have little pros and many cons. I also believe token replacement is a great way to take care of the information hiding issues.

--
Wil Moore III

Best Practices for Working with Open-Source Developers
http://www.faqs.org/docs/artu/ch19s02.html

Why is Bottom-posting better than Top-posting:
http://www.caliburn.nl/topposting.html

DO NOT TOP-POST and DO trim your replies:
http://linux.sgms-centre.com/misc/netiquette.php#toppost
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: Glob path config merging

EvanDotPro
In reply to this post by EvanDotPro
Hi all,

Just wanted to point out that I've added this as a topic on the agenda
for our meeting tomorrow:

http://framework.zend.com/wiki/display/ZFDEV2/2011-12-07+Meeting+Agenda

So if you have any ideas, let's hear them now, or be prepared to bring
them up in the meeting tomorrow!

--
Evan Coury

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Loading...