Benchmarks (yes, again)

classic Classic list List threaded Threaded
7 messages Options
Reply | Threaded
Open this post in threaded view
|

Benchmarks (yes, again)

Marco Pivetta
I'm hereby asking for reviews so that we can carefully verify that I didn't forget anything relevant.

Also, I will need to setup an ORM version of the tests later on.

Regardless if the results will be good or bad, I hope this will put a full stop on any useless discussion about who's faster (if that even makes any sense :) )

Thanks!
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarks (yes, again)

weierophinney
Administrator
Enrico has done extensive benchmarking of the tutorial application.
One thing we noticed is that for ZF1 applications, the more modules
you add, the slower the application becomes. With ZF2, performance
follows a logarithmic curve, whereby performance largely stays the
same as you add more modules.

My point being: I'm not particularly concerned with benchmarks for
"Hello World", as I feel they don't give an accurate idea overall of
the application performance. While I understand the arguments of "this
is the best performance you'll get from your framework; doing anything
else will typically decrease performance," I think they miss huge
swathes of application architecture. A good application will often
generate static HTML pages, so that huge portions of the site, while
originally part of the application, are static. Others will use
caching such as memcached or redis, and bail early in the request when
a cache hit is discovered.

If you are using strategies such as this, these "hello world"
benchmarks are entirely irrelevant; the framework handles the heavy
hitting stuff only - where things like I/O are going to be the biggest
bottleneck anyways.

On Sun, Apr 28, 2013 at 3:53 PM, Marco Pivetta <[hidden email]> wrote:

> Heya!
>
> I just submitted https://github.com/TechEmpower/FrameworkBenchmarks/pull/218
> to get Zend Framework 2 listed on http://www.techempower.com/benchmarks/
>
> I'm hereby asking for reviews so that we can carefully verify that I didn't
> forget anything relevant.
>
> Also, I will need to setup an ORM version of the tests later on.
>
> Regardless if the results will be good or bad, I hope this will put a full
> stop on any useless discussion about who's faster (if that even makes any
> sense :) )
>
> Thanks!
>
>
> Marco Pivetta
>
> http://twitter.com/Ocramius
>
> http://ocramius.github.com/



--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarks (yes, again)

Marco Pivetta
Hey Matthew,

Yes, I don't care about hello-world-ish applications either, and I understand (and share) all your thoughts on how application architecture affects a final product.

There's a "but": so far there's only been only bashing against ZF2, without proper testing and proper collaboration from the community.
So far we only had the typical blogposts about one guy posting a benchmark just to collect a few visitors for his blog.
By having a benchmark there (maintained by the community) we just have a test bench that will push people toward optimizing baseline performance where possible.

Take it as "carrot and stick" approach for fans of optimization.

There's still people out there that believe there's any meaning in optimizing Gentoo's performance (not me),  and even if it's quite useless, it helps a little bit on the long run. 5% speedup done by <random guy> is always good if it comes for free, no? ;)




On 29 April 2013 15:40, Matthew Weier O'Phinney <[hidden email]> wrote:
Enrico has done extensive benchmarking of the tutorial application.
One thing we noticed is that for ZF1 applications, the more modules
you add, the slower the application becomes. With ZF2, performance
follows a logarithmic curve, whereby performance largely stays the
same as you add more modules.

My point being: I'm not particularly concerned with benchmarks for
"Hello World", as I feel they don't give an accurate idea overall of
the application performance. While I understand the arguments of "this
is the best performance you'll get from your framework; doing anything
else will typically decrease performance," I think they miss huge
swathes of application architecture. A good application will often
generate static HTML pages, so that huge portions of the site, while
originally part of the application, are static. Others will use
caching such as memcached or redis, and bail early in the request when
a cache hit is discovered.

If you are using strategies such as this, these "hello world"
benchmarks are entirely irrelevant; the framework handles the heavy
hitting stuff only - where things like I/O are going to be the biggest
bottleneck anyways.

On Sun, Apr 28, 2013 at 3:53 PM, Marco Pivetta <[hidden email]> wrote:
> Heya!
>
> I just submitted https://github.com/TechEmpower/FrameworkBenchmarks/pull/218
> to get Zend Framework 2 listed on http://www.techempower.com/benchmarks/
>
> I'm hereby asking for reviews so that we can carefully verify that I didn't
> forget anything relevant.
>
> Also, I will need to setup an ORM version of the tests later on.
>
> Regardless if the results will be good or bad, I hope this will put a full
> stop on any useless discussion about who's faster (if that even makes any
> sense :) )
>
> Thanks!
>
>
> Marco Pivetta
>
> http://twitter.com/Ocramius
>
> http://ocramius.github.com/



--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

Reply | Threaded
Open this post in threaded view
|

Re: Benchmarks (yes, again)

Enrico Zimuel-2
Hi Marco and Matthew,

i agree with both of you, we need to educate people to understand what means use a framework and how to compare real applications and not just "hello world" examples. At the same time I think we can easily participate in public benchmarks like the one proposed by Marco and be sure to point people to more relevant benchmarks like the one we want to publish.
I only need to re-run the benchmarks on the last version of ZF2 using the last stable version of PHP and I will publish the results. We compared ZF2 (with Zend\Db and Doctrine) with ZF1 (with Zend_Db and Doctrine) and Symfony2, using a simple CRUD application.

Enrico


On Mon, Apr 29, 2013 at 4:33 PM, Marco Pivetta <[hidden email]> wrote:
Hey Matthew,

Yes, I don't care about hello-world-ish applications either, and I understand (and share) all your thoughts on how application architecture affects a final product.

There's a "but": so far there's only been only bashing against ZF2, without proper testing and proper collaboration from the community.
So far we only had the typical blogposts about one guy posting a benchmark just to collect a few visitors for his blog.
By having a benchmark there (maintained by the community) we just have a test bench that will push people toward optimizing baseline performance where possible.

Take it as "carrot and stick" approach for fans of optimization.

There's still people out there that believe there's any meaning in optimizing Gentoo's performance (not me),  and even if it's quite useless, it helps a little bit on the long run. 5% speedup done by <random guy> is always good if it comes for free, no? ;)

On 29 April 2013 15:40, Matthew Weier O'Phinney <[hidden email]> wrote:
Enrico has done extensive benchmarking of the tutorial application.
One thing we noticed is that for ZF1 applications, the more modules
you add, the slower the application becomes. With ZF2, performance
follows a logarithmic curve, whereby performance largely stays the
same as you add more modules.

My point being: I'm not particularly concerned with benchmarks for
"Hello World", as I feel they don't give an accurate idea overall of
the application performance. While I understand the arguments of "this
is the best performance you'll get from your framework; doing anything
else will typically decrease performance," I think they miss huge
swathes of application architecture. A good application will often
generate static HTML pages, so that huge portions of the site, while
originally part of the application, are static. Others will use
caching such as memcached or redis, and bail early in the request when
a cache hit is discovered.

If you are using strategies such as this, these "hello world"
benchmarks are entirely irrelevant; the framework handles the heavy
hitting stuff only - where things like I/O are going to be the biggest
bottleneck anyways.

On Sun, Apr 28, 2013 at 3:53 PM, Marco Pivetta <[hidden email]> wrote:
> Heya!
>
> I just submitted https://github.com/TechEmpower/FrameworkBenchmarks/pull/218
> to get Zend Framework 2 listed on http://www.techempower.com/benchmarks/
>
> I'm hereby asking for reviews so that we can carefully verify that I didn't
> forget anything relevant.
>
> Also, I will need to setup an ORM version of the tests later on.
>
> Regardless if the results will be good or bad, I hope this will put a full
> stop on any useless discussion about who's faster (if that even makes any
> sense :) )
>
> Thanks!
>
>
> Marco Pivetta
>
> http://twitter.com/Ocramius
>
> http://ocramius.github.com/



--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc




--
Enrico Zimuel
Senior PHP Engineer     | [hidden email]
Zend Framework Team     | http://framework.zend.com
Zend Technologies Ltd.
http://www.zend.com
Reply | Threaded
Open this post in threaded view
|

Re: Benchmarks (yes, again)

Tomáš Fejfar
The thing is that you need to do most of the optimalization yourself. There is no "one-click" optimalization as you can see with symfony included. That's why most people care about "simple benchmarks". They show what you can get for free. You can memcache anything, drop ORM and write optimal queries, but that all requires additional work. Some frameworks are fast enough in baseline, so that you never even need to get to memcache. 

Tomas Fejfar, w3w.cz

twitter: @tomasfejfar
email: [hidden email]
www: http://www.tomasfejfar.cz



On Mon, Apr 29, 2013 at 4:41 PM, Enrico Zimuel <[hidden email]> wrote:
Hi Marco and Matthew,

i agree with both of you, we need to educate people to understand what means use a framework and how to compare real applications and not just "hello world" examples. At the same time I think we can easily participate in public benchmarks like the one proposed by Marco and be sure to point people to more relevant benchmarks like the one we want to publish.
I only need to re-run the benchmarks on the last version of ZF2 using the last stable version of PHP and I will publish the results. We compared ZF2 (with Zend\Db and Doctrine) with ZF1 (with Zend_Db and Doctrine) and Symfony2, using a simple CRUD application.

Enrico


On Mon, Apr 29, 2013 at 4:33 PM, Marco Pivetta <[hidden email]> wrote:
Hey Matthew,

Yes, I don't care about hello-world-ish applications either, and I understand (and share) all your thoughts on how application architecture affects a final product.

There's a "but": so far there's only been only bashing against ZF2, without proper testing and proper collaboration from the community.
So far we only had the typical blogposts about one guy posting a benchmark just to collect a few visitors for his blog.
By having a benchmark there (maintained by the community) we just have a test bench that will push people toward optimizing baseline performance where possible.

Take it as "carrot and stick" approach for fans of optimization.

There's still people out there that believe there's any meaning in optimizing Gentoo's performance (not me),  and even if it's quite useless, it helps a little bit on the long run. 5% speedup done by <random guy> is always good if it comes for free, no? ;)

On 29 April 2013 15:40, Matthew Weier O'Phinney <[hidden email]> wrote:
Enrico has done extensive benchmarking of the tutorial application.
One thing we noticed is that for ZF1 applications, the more modules
you add, the slower the application becomes. With ZF2, performance
follows a logarithmic curve, whereby performance largely stays the
same as you add more modules.

My point being: I'm not particularly concerned with benchmarks for
"Hello World", as I feel they don't give an accurate idea overall of
the application performance. While I understand the arguments of "this
is the best performance you'll get from your framework; doing anything
else will typically decrease performance," I think they miss huge
swathes of application architecture. A good application will often
generate static HTML pages, so that huge portions of the site, while
originally part of the application, are static. Others will use
caching such as memcached or redis, and bail early in the request when
a cache hit is discovered.

If you are using strategies such as this, these "hello world"
benchmarks are entirely irrelevant; the framework handles the heavy
hitting stuff only - where things like I/O are going to be the biggest
bottleneck anyways.

On Sun, Apr 28, 2013 at 3:53 PM, Marco Pivetta <[hidden email]> wrote:
> Heya!
>
> I just submitted https://github.com/TechEmpower/FrameworkBenchmarks/pull/218
> to get Zend Framework 2 listed on http://www.techempower.com/benchmarks/
>
> I'm hereby asking for reviews so that we can carefully verify that I didn't
> forget anything relevant.
>
> Also, I will need to setup an ORM version of the tests later on.
>
> Regardless if the results will be good or bad, I hope this will put a full
> stop on any useless discussion about who's faster (if that even makes any
> sense :) )
>
> Thanks!
>
>
> Marco Pivetta
>
> http://twitter.com/Ocramius
>
> http://ocramius.github.com/



--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc




--
Enrico Zimuel
Senior PHP Engineer     | [hidden email]
Zend Framework Team     | http://framework.zend.com
Zend Technologies Ltd.
http://www.zend.com

Reply | Threaded
Open this post in threaded view
|

Re: Benchmarks (yes, again)

Marco Pivetta
That is true only for small applications under low load, Tomáš. That's basics of computational complexity: you cannot fight load just by removing linear overhead ;)



On 29 April 2013 19:13, Tomáš Fejfar <[hidden email]> wrote:
The thing is that you need to do most of the optimalization yourself. There is no "one-click" optimalization as you can see with symfony included. That's why most people care about "simple benchmarks". They show what you can get for free. You can memcache anything, drop ORM and write optimal queries, but that all requires additional work. Some frameworks are fast enough in baseline, so that you never even need to get to memcache. 

Tomas Fejfar, w3w.cz

twitter: @tomasfejfar
email: [hidden email]
www: http://www.tomasfejfar.cz



On Mon, Apr 29, 2013 at 4:41 PM, Enrico Zimuel <[hidden email]> wrote:
Hi Marco and Matthew,

i agree with both of you, we need to educate people to understand what means use a framework and how to compare real applications and not just "hello world" examples. At the same time I think we can easily participate in public benchmarks like the one proposed by Marco and be sure to point people to more relevant benchmarks like the one we want to publish.
I only need to re-run the benchmarks on the last version of ZF2 using the last stable version of PHP and I will publish the results. We compared ZF2 (with Zend\Db and Doctrine) with ZF1 (with Zend_Db and Doctrine) and Symfony2, using a simple CRUD application.

Enrico


On Mon, Apr 29, 2013 at 4:33 PM, Marco Pivetta <[hidden email]> wrote:
Hey Matthew,

Yes, I don't care about hello-world-ish applications either, and I understand (and share) all your thoughts on how application architecture affects a final product.

There's a "but": so far there's only been only bashing against ZF2, without proper testing and proper collaboration from the community.
So far we only had the typical blogposts about one guy posting a benchmark just to collect a few visitors for his blog.
By having a benchmark there (maintained by the community) we just have a test bench that will push people toward optimizing baseline performance where possible.

Take it as "carrot and stick" approach for fans of optimization.

There's still people out there that believe there's any meaning in optimizing Gentoo's performance (not me),  and even if it's quite useless, it helps a little bit on the long run. 5% speedup done by <random guy> is always good if it comes for free, no? ;)

On 29 April 2013 15:40, Matthew Weier O'Phinney <[hidden email]> wrote:
Enrico has done extensive benchmarking of the tutorial application.
One thing we noticed is that for ZF1 applications, the more modules
you add, the slower the application becomes. With ZF2, performance
follows a logarithmic curve, whereby performance largely stays the
same as you add more modules.

My point being: I'm not particularly concerned with benchmarks for
"Hello World", as I feel they don't give an accurate idea overall of
the application performance. While I understand the arguments of "this
is the best performance you'll get from your framework; doing anything
else will typically decrease performance," I think they miss huge
swathes of application architecture. A good application will often
generate static HTML pages, so that huge portions of the site, while
originally part of the application, are static. Others will use
caching such as memcached or redis, and bail early in the request when
a cache hit is discovered.

If you are using strategies such as this, these "hello world"
benchmarks are entirely irrelevant; the framework handles the heavy
hitting stuff only - where things like I/O are going to be the biggest
bottleneck anyways.

On Sun, Apr 28, 2013 at 3:53 PM, Marco Pivetta <[hidden email]> wrote:
> Heya!
>
> I just submitted https://github.com/TechEmpower/FrameworkBenchmarks/pull/218
> to get Zend Framework 2 listed on http://www.techempower.com/benchmarks/
>
> I'm hereby asking for reviews so that we can carefully verify that I didn't
> forget anything relevant.
>
> Also, I will need to setup an ORM version of the tests later on.
>
> Regardless if the results will be good or bad, I hope this will put a full
> stop on any useless discussion about who's faster (if that even makes any
> sense :) )
>
> Thanks!
>
>
> Marco Pivetta
>
> http://twitter.com/Ocramius
>
> http://ocramius.github.com/



--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc




--
Enrico Zimuel
Senior PHP Engineer     | [hidden email]
Zend Framework Team     | http://framework.zend.com
Zend Technologies Ltd.
http://www.zend.com


Reply | Threaded
Open this post in threaded view
|

Re: Benchmarks (yes, again)

Tomáš Fejfar
But that's what majority uses frameworks for! :) Until some level of traffic (quite a fair amount, which most apps in ZF won't get over) you can get well with faster baseline. Those are the 80% that care about benchmarks. The rest either know they won't get along without massive optimization from the start or are enterprise enough to have the "throw more RAM/CPU at it" mentality and don't care about speed...

Tomas Fejfar 
tomasfejfar.cz



On Mon, Apr 29, 2013 at 7:15 PM, Marco Pivetta <[hidden email]> wrote:
That is true only for small applications under low load, Tomáš. That's basics of computational complexity: you cannot fight load just by removing linear overhead ;)



On 29 April 2013 19:13, Tomáš Fejfar <[hidden email]> wrote:
The thing is that you need to do most of the optimalization yourself. There is no "one-click" optimalization as you can see with symfony included. That's why most people care about "simple benchmarks". They show what you can get for free. You can memcache anything, drop ORM and write optimal queries, but that all requires additional work. Some frameworks are fast enough in baseline, so that you never even need to get to memcache. 

Tomas Fejfar, w3w.cz

twitter: @tomasfejfar
email: [hidden email]
www: http://www.tomasfejfar.cz



On Mon, Apr 29, 2013 at 4:41 PM, Enrico Zimuel <[hidden email]> wrote:
Hi Marco and Matthew,

i agree with both of you, we need to educate people to understand what means use a framework and how to compare real applications and not just "hello world" examples. At the same time I think we can easily participate in public benchmarks like the one proposed by Marco and be sure to point people to more relevant benchmarks like the one we want to publish.
I only need to re-run the benchmarks on the last version of ZF2 using the last stable version of PHP and I will publish the results. We compared ZF2 (with Zend\Db and Doctrine) with ZF1 (with Zend_Db and Doctrine) and Symfony2, using a simple CRUD application.

Enrico


On Mon, Apr 29, 2013 at 4:33 PM, Marco Pivetta <[hidden email]> wrote:
Hey Matthew,

Yes, I don't care about hello-world-ish applications either, and I understand (and share) all your thoughts on how application architecture affects a final product.

There's a "but": so far there's only been only bashing against ZF2, without proper testing and proper collaboration from the community.
So far we only had the typical blogposts about one guy posting a benchmark just to collect a few visitors for his blog.
By having a benchmark there (maintained by the community) we just have a test bench that will push people toward optimizing baseline performance where possible.

Take it as "carrot and stick" approach for fans of optimization.

There's still people out there that believe there's any meaning in optimizing Gentoo's performance (not me),  and even if it's quite useless, it helps a little bit on the long run. 5% speedup done by <random guy> is always good if it comes for free, no? ;)

On 29 April 2013 15:40, Matthew Weier O'Phinney <[hidden email]> wrote:
Enrico has done extensive benchmarking of the tutorial application.
One thing we noticed is that for ZF1 applications, the more modules
you add, the slower the application becomes. With ZF2, performance
follows a logarithmic curve, whereby performance largely stays the
same as you add more modules.

My point being: I'm not particularly concerned with benchmarks for
"Hello World", as I feel they don't give an accurate idea overall of
the application performance. While I understand the arguments of "this
is the best performance you'll get from your framework; doing anything
else will typically decrease performance," I think they miss huge
swathes of application architecture. A good application will often
generate static HTML pages, so that huge portions of the site, while
originally part of the application, are static. Others will use
caching such as memcached or redis, and bail early in the request when
a cache hit is discovered.

If you are using strategies such as this, these "hello world"
benchmarks are entirely irrelevant; the framework handles the heavy
hitting stuff only - where things like I/O are going to be the biggest
bottleneck anyways.

On Sun, Apr 28, 2013 at 3:53 PM, Marco Pivetta <[hidden email]> wrote:
> Heya!
>
> I just submitted https://github.com/TechEmpower/FrameworkBenchmarks/pull/218
> to get Zend Framework 2 listed on http://www.techempower.com/benchmarks/
>
> I'm hereby asking for reviews so that we can carefully verify that I didn't
> forget anything relevant.
>
> Also, I will need to setup an ORM version of the tests later on.
>
> Regardless if the results will be good or bad, I hope this will put a full
> stop on any useless discussion about who's faster (if that even makes any
> sense :) )
>
> Thanks!
>
>
> Marco Pivetta
>
> http://twitter.com/Ocramius
>
> http://ocramius.github.com/



--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc




--

Enrico Zimuel
Senior PHP Engineer     | [hidden email]
Zend Framework Team     | http://framework.zend.com
Zend Technologies Ltd.
http://www.zend.com