Quantcast

Fixing Zend_Db_Statement::_stripQuoted() seg-faults on large SQL strings in ZF 1.x

classic Classic list List threaded Threaded
4 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Fixing Zend_Db_Statement::_stripQuoted() seg-faults on large SQL strings in ZF 1.x

ralphschindler
Hi all,

In the past we've been reluctant to solve an issue inside
Zend_Db_Statement::_stripQuoted() that caused PHP's PCRE to seg-fault
and crash.

http://framework.zend.com/issues/browse/ZF-5063
http://framework.zend.com/issues/browse/ZF-10209

We've been reluctant for a number of reasons:

   * the problem is not directly in ZF, but in PHP's PCRE
   * the problem is only exposed on certain architectures
   * fixing the problem might cause unintended BC issues

That said, given the amount of research a number of contributors have
put into the matter, primarily by Bart McLeod, we feel comfortable
moving forward with this fix for ZF 1.12 as it gives developers some
flexibility in avoiding the problem, and the fix is fairly well tested
against BC issues.

I will let Bart get into the complexities of the fix, but I would like
everyone to test what is currently in _trunk_ if you have large SQL that
is being passed through Zend_Db.

Thanks!
Ralph Schindler

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Fixing Zend_Db_Statement::_stripQuoted() seg-faults on large SQL strings in ZF 1.x

Bart McLeod
OK,

Let me follow up with the details in case you are interested to contribute by testing the fix on your system against your database. This is important, we need all the help we can get!

I can reproduce issue ZF-5063 with the following code:

       $db = Zend_Db_Table::getDefaultAdapter();
       $value = str_repeat('a', 66000) . 'x'; // 66,000 times an 'a' with an 'x' appended to it, to identify the end.
       $query = "INSERT INTO `pcre` (`test`) VALUES ('$value')";

In my case, the default adapter was mysql and I am on a 64bit system.

You might get a segmentation fault from this, which you can see in you apache error log. The code snippet was inside a controller action of a default Zend Framework project (see the full code below).

To reproduce with this code, you need a database connection and a table named 'pcre' with a field named 'test' (of type LONG_TEXT in mysql). In my case, this was the only field.

**Fix the segfault in case you get it**

In case you also reproduce the segfault, you can add a setting to php.ini to modify the behavior of the PCRE library, that is causing the segfault:

[PCRE]
pcre.recursion_limit = 1000

If you set this, the segfault should dissapear. The default pcre.recursion_limit = 100000, which is too high, apparently.
If you set it to a value that is simply to low, you will get many errors accross many components.
If you have or get idea's on what an optimum default setting would be, please let us know.

Instead of the segfault, you will either get a memory_limit error, which is ok, since you can increase the memory_limit if you need to, or you will get a database error because you query exceeds the maximum size, in case that is too low. You can play with how much you allow as the maximum packet size for your database and the memory_limit in php, but in the end, it is important that **you do no longer get the segmentation fault**. If you don't and you did before, you have found a working setting for pcre.recursion_limit.

The query above is a very simple query. We would like to know if real life queries that previously caused a segfault, will work now, with the new implementation and optionally a different recursion_limit on PCRE.

I intend to change the current implementation of Zend_Db_Statement so that it sets a sensible default for pcre.recursion_limit at runtime and restores the original limit afterwards. In addition, I would like to make this configurable through the adapter configuration, so that you can change the default if it not right for a specific use case you may have.

**Full controller code:**

/**
 * DbController
 */
class DbController extends Zend_Controller_Action
{
    //  http://localhost/db/st
    public function stAction()
    {
       $db = Zend_Db_Table::getDefaultAdapter();
       $value = str_repeat('a', 66000) . 'x';
       $query = "INSERT INTO `pcre` (`test`) VALUES ('$value')";
       $this->view->query = $query;
    }
}




Op 21-05-12 15:59, Ralph Schindler schreef:
Hi all,

In the past we've been reluctant to solve an issue inside Zend_Db_Statement::_stripQuoted() that caused PHP's PCRE to seg-fault and crash.

http://framework.zend.com/issues/browse/ZF-5063
http://framework.zend.com/issues/browse/ZF-10209

We've been reluctant for a number of reasons:

  * the problem is not directly in ZF, but in PHP's PCRE
  * the problem is only exposed on certain architectures
  * fixing the problem might cause unintended BC issues

That said, given the amount of research a number of contributors have put into the matter, primarily by Bart McLeod, we feel comfortable moving forward with this fix for ZF 1.12 as it gives developers some flexibility in avoiding the problem, and the fix is fairly well tested against BC issues.

I will let Bart get into the complexities of the fix, but I would like everyone to test what is currently in _trunk_ if you have large SQL that is being passed through Zend_Db.

Thanks!
Ralph Schindler


--
Bart McLeod
Space Web
Middenlaan 47
6869 VN Heveadorp
The Netherlands
t +31(0)26 3392952
m 06 51 51 89 71
@ [hidden email]
www.spaceweb.nl
zce PHP 4 logo zce PHP 5 logo zce PHP 5.3 logo zce Zend Framework logo

Bart McLeod is a Zend Certified Engineer.

Click to verify!

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Fixing Zend_Db_Statement::_stripQuoted() seg-faults on large SQL strings in ZF 1.x

Bart McLeod
I just noticed I omitted one essential line in the reproduction code:

       $db = Zend_Db_Table::getDefaultAdapter();
       $value = str_repeat('a', 66000) . 'x';
       $query = "INSERT INTO `pcre` (`test`) VALUES ('$value')";
       $this->view->query = $query;
       var_dump($db->query($query)); // added this line, that actually triggers the segfault, if it exists



Op 21-05-12 16:42, Bart McLeod schreef:
OK,

Let me follow up with the details in case you are interested to contribute by testing the fix on your system against your database. This is important, we need all the help we can get!

I can reproduce issue ZF-5063 with the following code:

       $db = Zend_Db_Table::getDefaultAdapter();
       $value = str_repeat('a', 66000) . 'x'; // 66,000 times an 'a' with an 'x' appended to it, to identify the end.
       $query = "INSERT INTO `pcre` (`test`) VALUES ('$value')";

In my case, the default adapter was mysql and I am on a 64bit system.

You might get a segmentation fault from this, which you can see in you apache error log. The code snippet was inside a controller action of a default Zend Framework project (see the full code below).

To reproduce with this code, you need a database connection and a table named 'pcre' with a field named 'test' (of type LONG_TEXT in mysql). In my case, this was the only field.

**Fix the segfault in case you get it**

In case you also reproduce the segfault, you can add a setting to php.ini to modify the behavior of the PCRE library, that is causing the segfault:

[PCRE]
pcre.recursion_limit = 1000

If you set this, the segfault should dissapear. The default pcre.recursion_limit = 100000, which is too high, apparently.
If you set it to a value that is simply to low, you will get many errors accross many components.
If you have or get idea's on what an optimum default setting would be, please let us know.

Instead of the segfault, you will either get a memory_limit error, which is ok, since you can increase the memory_limit if you need to, or you will get a database error because you query exceeds the maximum size, in case that is too low. You can play with how much you allow as the maximum packet size for your database and the memory_limit in php, but in the end, it is important that **you do no longer get the segmentation fault**. If you don't and you did before, you have found a working setting for pcre.recursion_limit.

The query above is a very simple query. We would like to know if real life queries that previously caused a segfault, will work now, with the new implementation and optionally a different recursion_limit on PCRE.

I intend to change the current implementation of Zend_Db_Statement so that it sets a sensible default for pcre.recursion_limit at runtime and restores the original limit afterwards. In addition, I would like to make this configurable through the adapter configuration, so that you can change the default if it not right for a specific use case you may have.

**Full controller code:**

/**
 * DbController
 */
class DbController extends Zend_Controller_Action
{
    //  http://localhost/db/st
    public function stAction()
    {
       $db = Zend_Db_Table::getDefaultAdapter();
       $value = str_repeat('a', 66000) . 'x';
       $query = "INSERT INTO `pcre` (`test`) VALUES ('$value')";
       $this->view->query = $query;
    }
}




Op 21-05-12 15:59, Ralph Schindler schreef:
Hi all,

In the past we've been reluctant to solve an issue inside Zend_Db_Statement::_stripQuoted() that caused PHP's PCRE to seg-fault and crash.

http://framework.zend.com/issues/browse/ZF-5063
http://framework.zend.com/issues/browse/ZF-10209

We've been reluctant for a number of reasons:

  * the problem is not directly in ZF, but in PHP's PCRE
  * the problem is only exposed on certain architectures
  * fixing the problem might cause unintended BC issues

That said, given the amount of research a number of contributors have put into the matter, primarily by Bart McLeod, we feel comfortable moving forward with this fix for ZF 1.12 as it gives developers some flexibility in avoiding the problem, and the fix is fairly well tested against BC issues.

I will let Bart get into the complexities of the fix, but I would like everyone to test what is currently in _trunk_ if you have large SQL that is being passed through Zend_Db.

Thanks!
Ralph Schindler


--
Bart McLeod
Space Web
Middenlaan 47
6869 VN Heveadorp
The Netherlands
t +31(0)26 3392952
m 06 51 51 89 71
@ [hidden email]
www.spaceweb.nl
zce
                  PHP 4 logo zce
                  PHP 5 logo zce
                  PHP 5.3 logo zce
                  Zend Framework logo

Bart McLeod is a Zend Certified Engineer.

Click to verify!


--
Bart McLeod
Space Web
Middenlaan 47
6869 VN Heveadorp
The Netherlands
t +31(0)26 3392952
m 06 51 51 89 71
@ [hidden email]
www.spaceweb.nl
zce PHP 4 logo zce PHP 5 logo zce PHP 5.3 logo zce Zend Framework logo

Bart McLeod is a Zend Certified Engineer.

Click to verify!

Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate
star

Re: Fixing Zend_Db_Statement::_stripQuoted() seg-faults on large SQL strings in ZF 1.x

Colin Guthrie-6
Hi Guys,


Sorry for resurrecting an old thread but I've just updated to 1.12.1 and
am now being bitten by the changes made in regard to this problem.

I've distilled my problem down to this commit:
http://framework.zend.com/code/revision.php?repname=Zend+Framework&path=%2Ftrunk%2Flibrary%2FZend%2FDb%2FStatement.php&rev=24789&peg=24789

Reverting it seemed to fix things.

I've got a wrapper class around extended multi-row insert syntax. In
order to prevent it using too much memory it limits it's queries to 32k
chars for each insert.

Sadly with the above commit, it causes PHP to segv due to it seems
recursion in libpcre.so.1 (10.8k calls in the gdb dump!).

My data is heavily quoted (lots of json and/or serialized data) and all
quoting is already done when passing the query to the ->query() call.

I'll try and come up with a simplified reproduction case and open a new bug.

Just thought I'd send a headsup to the list first.

Col


'Twas brillig, and Bart McLeod at 21/05/12 20:35 did gyre and gimble:

> I just noticed I omitted one essential line in the reproduction code:
>
>        $db = Zend_Db_Table::getDefaultAdapter();
>        $value = str_repeat('a', 66000) . 'x';
>        $query = "INSERT INTO `pcre` (`test`) VALUES ('$value')";
>        $this->view->query = $query;
>        var_dump($db->query($query)); // added this line, that actually
> triggers the segfault, if it exists
>
>
>
> Op 21-05-12 16:42, Bart McLeod schreef:
>> OK,
>>
>> Let me follow up with the details in case you are interested to
>> contribute by testing the fix on your system against your database.
>> This is important, we need all the help we can get!
>>
>> I can reproduce issue ZF-5063 with the following code:
>>
>>        $db = Zend_Db_Table::getDefaultAdapter();
>>        $value = str_repeat('a', 66000) . 'x'; // 66,000 times an 'a'
>> with an 'x' appended to it, to identify the end.
>>        $query = "INSERT INTO `pcre` (`test`) VALUES ('$value')";
>>
>> In my case, the default adapter was mysql and I am on a 64bit system.
>>
>> You might get a segmentation fault from this, which you can see in you
>> apache error log. The code snippet was inside a controller action of a
>> default Zend Framework project (see the full code below).
>>
>> To reproduce with this code, you need a database connection and a
>> table named 'pcre' with a field named 'test' (of type LONG_TEXT in
>> mysql). In my case, this was the only field.
>>
>> **Fix the segfault in case you get it**
>>
>> In case you also reproduce the segfault, you can add a setting to
>> php.ini to modify the behavior of the PCRE library, that is causing
>> the segfault:
>>
>> [PCRE]
>> pcre.recursion_limit = 1000
>>
>> If you set this, the segfault should dissapear. The default
>> pcre.recursion_limit = 100000, which is too high, apparently.
>> If you set it to a value that is simply to low, you will get many
>> errors accross many components.
>> If you have or get idea's on what an optimum default setting would be,
>> please let us know.
>>
>> Instead of the segfault, you will either get a memory_limit error,
>> which is ok, since you can increase the memory_limit if you need to,
>> or you will get a database error because you query exceeds the maximum
>> size, in case that is too low. You can play with how much you allow as
>> the maximum packet size for your database and the memory_limit in php,
>> but in the end, it is important that **you do no longer get the
>> segmentation fault**. If you don't and you did before, you have found
>> a working setting for pcre.recursion_limit.
>>
>> The query above is a very simple query. We would like to know if real
>> life queries that previously caused a segfault, will work now, with
>> the new implementation and optionally a different recursion_limit on PCRE.
>>
>> I intend to change the current implementation of Zend_Db_Statement so
>> that it sets a sensible default for pcre.recursion_limit at runtime
>> and restores the original limit afterwards. In addition, I would like
>> to make this configurable through the adapter configuration, so that
>> you can change the default if it not right for a specific use case you
>> may have.
>>
>> **Full controller code:**
>>
>> /**
>>  * DbController
>>  */
>> class DbController extends Zend_Controller_Action
>> {
>>     //  http://localhost/db/st
>>     public function stAction()
>>     {
>>        $db = Zend_Db_Table::getDefaultAdapter();
>>        $value = str_repeat('a', 66000) . 'x';
>>        $query = "INSERT INTO `pcre` (`test`) VALUES ('$value')";
>>        $this->view->query = $query;
>>     }
>> }
>>
>>
>>
>>
>> Op 21-05-12 15:59, Ralph Schindler schreef:
>>> Hi all,
>>>
>>> In the past we've been reluctant to solve an issue inside
>>> Zend_Db_Statement::_stripQuoted() that caused PHP's PCRE to seg-fault
>>> and crash.
>>>
>>> http://framework.zend.com/issues/browse/ZF-5063
>>> http://framework.zend.com/issues/browse/ZF-10209
>>>
>>> We've been reluctant for a number of reasons:
>>>
>>>   * the problem is not directly in ZF, but in PHP's PCRE
>>>   * the problem is only exposed on certain architectures
>>>   * fixing the problem might cause unintended BC issues
>>>
>>> That said, given the amount of research a number of contributors have
>>> put into the matter, primarily by Bart McLeod, we feel comfortable
>>> moving forward with this fix for ZF 1.12 as it gives developers some
>>> flexibility in avoiding the problem, and the fix is fairly well
>>> tested against BC issues.
>>>
>>> I will let Bart get into the complexities of the fix, but I would
>>> like everyone to test what is currently in _trunk_ if you have large
>>> SQL that is being passed through Zend_Db.
>>>
>>> Thanks!
>>> Ralph Schindler
>>>
>>
>> --
>> Bart McLeod
>> *Space Web*
>> Middenlaan 47
>> 6869 VN Heveadorp
>> The Netherlands
>> *t* +31(0)26 3392952
>> *m* 06 51 51 89 71
>> *@* [hidden email]
>> <mailto:[hidden email]>
>> www.spaceweb.nl <http://www.spaceweb.nl> zce PHP 4 logo zce PHP 5
>> logo zce PHP 5.3 logo zce Zend Framework logo
>>
>> /Bart McLeod is a Zend Certified Engineer./
>>
>> Click to verify! <http://www.zend.com/zce.php?c=ZEND004591&r=218204904>
>>
>>
>>
>
> --
> Bart McLeod
> *Space Web*
> Middenlaan 47
> 6869 VN Heveadorp
> The Netherlands
> *t* +31(0)26 3392952
> *m* 06 51 51 89 71
> *@* [hidden email]
> <mailto:[hidden email]>
> www.spaceweb.nl <http://www.spaceweb.nl> zce PHP 4 logo zce PHP 5 logo
> zce PHP 5.3 logo zce Zend Framework logo
>
> /Bart McLeod is a Zend Certified Engineer./
>
> Click to verify! <http://www.zend.com/zce.php?c=ZEND004591&r=218204904>
>
>
>


--

Colin Guthrie
gmane(at)colin.guthr.ie
http://colin.guthr.ie/

Day Job:
  Tribalogic Limited http://www.tribalogic.net/
Open Source:
  Mageia Contributor http://www.mageia.org/
  PulseAudio Hacker http://www.pulseaudio.org/
  Trac Hacker http://trac.edgewall.org/


--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Loading...