Sending http request with Transfer-Encoding: chunked

classic Classic list List threaded Threaded
6 messages Options
Reply | Threaded
Open this post in threaded view
|

Sending http request with Transfer-Encoding: chunked

Matt Kynx
Hi all,

I'm currently working on a stream wrapper for cloudfiles. It's based on Zend_Service_Amazon_E3_Stream but, instead of storing the files it's fetching / sending in memory, it only ever stores the current buffer. This should allow it to safely handle very large files.

Use cases include streaming a cloudfile straight to the user's browser, saving a file upload and copying files between Rackspace accounts.

The biggest challenges have been on the sending side. Zend_Http_Client_Adapter_Socket doesn't support sending with Transfer-Encoding: chunked. I've subclassed it and added methods for writing chunks to the socket. It works, but it's a bit of a hack. Zend_Http_Client::request() wants to read() from the adapter before I've sent all the chunks, so I've either got to fake a response just to keep it happy or subclass that too, with a lot of cut-and-pasting.

So I'm wondering, has anyone else has done anything similar? I saw a comment on the original Rackspace proposal indicating some work had been done on streaming [1].

If there's nothing out there already, I'd like to add my stuff directly to Zend_Http_Client_Adapter_Socket and modify Zend_Http_Client so writing the request and reading back from it are split into separate methods, with the current request() calling both. Would this need a proposal, or can I just submit it as a patch on the issue tracker?

Hopefully this will be useful to other people (particularly the E3 stream wrapper), so I'd like to keep it separate from my stream wrapper stuff. I'll post more on that when it's ready.

Regards,

Matt

Reply | Threaded
Open this post in threaded view
|

Re: Sending http request with Transfer-Encoding: chunked

Matt Kynx
Hi,

I've thrown up some code so you can see what I'm on about.

On Sat, Jun 30, 2012 at 1:27 PM, Matt Kynx <[hidden email]> wrote:

If there's nothing out there already, I'd like to add my stuff directly to Zend_Http_Client_Adapter_Socket and modify Zend_Http_Client so writing the request and reading back from it are split into separate methods, with the current request() calling both.

I've kept (most of) my changes to subclasses - on thinking about it, if no one else has needed this functionality it's better not to clutter up the main classes.

Hopefully this will be useful to other people (particularly the E3 stream wrapper), so I'd like to keep it separate from my stream wrapper stuff. I'll post more on that when it's ready.

This includes the changes to Zend_Service_Rackspace_Files to support streaming requests, a Zend_Services_Rackspace_Files_Stream wrapper class, plus some stuff for handle checksums on muliple-segment uploads. On the latter, I've still got to get uploads > 5G working in the main class. I also want a checksumming class that uses proc to calculate the checksums on the fly, rather than waiting for it when the upload is complete.

I'll post back when I've got some example code to show. If anyone's got any feedback it'd be greatly appreciated.

Regards,

Matt

Reply | Threaded
Open this post in threaded view
|

Re: Sending http request with Transfer-Encoding: chunked

Shahar Evron-2
I'm quite sure the re-implementation I started working on for the HTTP client & friends will work much better for your use case, with the abstracted entity class. However it's not going to be done for 2.0 as I was too slow about it. Maybe you'll want to use it as a drop-in replacement for your project.

See https://github.com/shevron/zf2/tree/feature/http-client-rewrite and my previous emails to this list about it. Catch me on IRC if you have questions or suggestions.

Shahar.

On Mon, Jul 2, 2012 at 12:10 PM, Matt Kynx <[hidden email]> wrote:
Hi,

I've thrown up some code so you can see what I'm on about.


On Sat, Jun 30, 2012 at 1:27 PM, Matt Kynx <[hidden email]> wrote:

If there's nothing out there already, I'd like to add my stuff directly to Zend_Http_Client_Adapter_Socket and modify Zend_Http_Client so writing the request and reading back from it are split into separate methods, with the current request() calling both.

I've kept (most of) my changes to subclasses - on thinking about it, if no one else has needed this functionality it's better not to clutter up the main classes.

Hopefully this will be useful to other people (particularly the E3 stream wrapper), so I'd like to keep it separate from my stream wrapper stuff. I'll post more on that when it's ready.

This includes the changes to Zend_Service_Rackspace_Files to support streaming requests, a Zend_Services_Rackspace_Files_Stream wrapper class, plus some stuff for handle checksums on muliple-segment uploads. On the latter, I've still got to get uploads > 5G working in the main class. I also want a checksumming class that uses proc to calculate the checksums on the fly, rather than waiting for it when the upload is complete.

I'll post back when I've got some example code to show. If anyone's got any feedback it'd be greatly appreciated.

Regards,

Matt


Reply | Threaded
Open this post in threaded view
|

Re: Sending http request with Transfer-Encoding: chunked

Matt Kynx
Hi Shahar,

On 2 Jul 2012, at 16:54, Shahar Evron <[hidden email]> wrote:

I'm quite sure the re-implementation I started working on for the HTTP client & friends will work much better for your use case, with the abstracted entity class. However it's not going to be done for 2.0 as I was too slow about it. Maybe you'll want to use it as a drop-in replacement for your project.

Coo, yes, much cleaner than zf1 :)

Unfortunately this is for a zf1 project I'm moving onto rackspace cloud. I'm planning to do a zf2 version once this is working, but I've got to get this going first. 


See https://github.com/shevron/zf2/tree/feature/http-client-rewrite and my previous emails to this list about it. Catch me on IRC if you have questions or suggestions.

The main issue I'm trying to solve with the http client subclass is to start the request going without trying to read a response back. Then I've got to write chunks to the socket as they come from the stream and only when they're all sent read the response. 

I've only had a quick look at your rewrite, but can't see support for this. Happy to look at adding it. 

But maybe I'm missing something obvious. That often happens ;)

Matt

Sent from the thing in my pocket


Shahar.

On Mon, Jul 2, 2012 at 12:10 PM, Matt Kynx <[hidden email]> wrote:
Hi,

I've thrown up some code so you can see what I'm on about.


On Sat, Jun 30, 2012 at 1:27 PM, Matt Kynx <[hidden email]> wrote:

If there's nothing out there already, I'd like to add my stuff directly to Zend_Http_Client_Adapter_Socket and modify Zend_Http_Client so writing the request and reading back from it are split into separate methods, with the current request() calling both.

I've kept (most of) my changes to subclasses - on thinking about it, if no one else has needed this functionality it's better not to clutter up the main classes.

Hopefully this will be useful to other people (particularly the E3 stream wrapper), so I'd like to keep it separate from my stream wrapper stuff. I'll post more on that when it's ready.

This includes the changes to Zend_Service_Rackspace_Files to support streaming requests, a Zend_Services_Rackspace_Files_Stream wrapper class, plus some stuff for handle checksums on muliple-segment uploads. On the latter, I've still got to get uploads > 5G working in the main class. I also want a checksumming class that uses proc to calculate the checksums on the fly, rather than waiting for it when the upload is complete.

I'll post back when I've got some example code to show. If anyone's got any feedback it'd be greatly appreciated.

Regards,

Matt


Reply | Threaded
Open this post in threaded view
|

Re: Sending http request with Transfer-Encoding: chunked

Shahar Evron-2

Coo, yes, much cleaner than zf1 :)

Unfortunately this is for a zf1 project I'm moving onto rackspace cloud. I'm planning to do a zf2 version once this is working, but I've got to get this
going first. 

Sorry, I didn't notice you were referring to zf1 ;)
 


See https://github.com/shevron/zf2/tree/feature/http-client-rewrite and my previous emails to this list about it. Catch me on IRC if you have questions or suggestions.

The main issue I'm trying to solve with the http client subclass is to start the request going without trying to read a response back. Then I've got to write chunks to the socket as they come from the stream and only when they're all sent read the response. 

I've only had a quick look at your rewrite, but can't see support for this. Happy to look at adding it. 

With my design you can create your own implementation of the Entity class. Entity classes represent the request body in an abstract form - all they need to implement (usually) is a 'read()' method which is called repeatedly by the transport layer until there's nothing more to read - so it is quite suitable for sending the request body as chunks as they are read from a stream.

Another plan I have (which was suggested here by others) was to hook an EventManager instance to the HTTP client and transport layer, and fire off events which you can hook into in certain points such as when the request is finished and before the response is read.

I might need to modify the API between the Entity and the Transport layer a bit to suite your use case, but I believe the implementation is quite close.

Shahar.

 

But maybe I'm missing something obvious. That often happens ;)

Matt

Sent from the thing in my pocket


Shahar.

On Mon, Jul 2, 2012 at 12:10 PM, Matt Kynx <[hidden email]> wrote:
Hi,

I've thrown up some code so you can see what I'm on about.


On Sat, Jun 30, 2012 at 1:27 PM, Matt Kynx <[hidden email]> wrote:

If there's nothing out there already, I'd like to add my stuff directly to Zend_Http_Client_Adapter_Socket and modify Zend_Http_Client so writing the request and reading back from it are split into separate methods, with the current request() calling both.

I've kept (most of) my changes to subclasses - on thinking about it, if no one else has needed this functionality it's better not to clutter up the main classes.

Hopefully this will be useful to other people (particularly the E3 stream wrapper), so I'd like to keep it separate from my stream wrapper stuff. I'll post more on that when it's ready.

This includes the changes to Zend_Service_Rackspace_Files to support streaming requests, a Zend_Services_Rackspace_Files_Stream wrapper class, plus some stuff for handle checksums on muliple-segment uploads. On the latter, I've still got to get uploads > 5G working in the main class. I also want a checksumming class that uses proc to calculate the checksums on the fly, rather than waiting for it when the upload is complete.

I'll post back when I've got some example code to show. If anyone's got any feedback it'd be greatly appreciated.

Regards,

Matt



Reply | Threaded
Open this post in threaded view
|

Re: Sending http request with Transfer-Encoding: chunked

Matt Kynx
Hi Shahar,

> With my design you can create your own implementation of the Entity class.
> Entity classes represent the request body in an abstract form - all they
> need to implement (usually) is a 'read()' method which is called repeatedly
> by the transport layer until there's nothing more to read - so it is quite
> suitable for sending the request body as chunks as they are read from a
> stream.

Yes, that does look like it would do the job. The one reservation I
have is that the send() method on the transport calls readResponse().
I know my use case is a little bit, um, special, but that wouldn't
work for me. Either sendRequest() and readResponse() would need to be
public or it should be up to the client to read the request back
(which is how it works in zf1). Or I subclass
Zend_Http_Transport_Socket, like I'm doing now. Or...

> Another plan I have (which was suggested here by others) was to hook an
> EventManager instance to the HTTP client and transport layer, and fire off
> events which you can hook into in certain points such as when the request is
> finished and before the response is read.

Hey, that would be very elegant. And could be good for lots of things
(thinking progress bars, 'bytes sent' displayed by a cli program,
etc).

Cheers,

Matt

>
> I might need to modify the API between the Entity and the Transport layer a
> bit to suite your use case, but I believe the implementation is quite close.
>
> Shahar.
>
>
>>
>>
>> But maybe I'm missing something obvious. That often happens ;)
>>
>> Matt
>>
>> Sent from the thing in my pocket
>>
>>
>> Shahar.
>>
>> On Mon, Jul 2, 2012 at 12:10 PM, Matt Kynx <[hidden email]> wrote:
>>>
>>> Hi,
>>>
>>> I've thrown up some code so you can see what I'm on about.
>>>
>>>
>>> On Sat, Jun 30, 2012 at 1:27 PM, Matt Kynx <[hidden email]> wrote:
>>>
>>>> If there's nothing out there already, I'd like to add my stuff directly
>>>> to Zend_Http_Client_Adapter_Socket and modify Zend_Http_Client so writing
>>>> the request and reading back from it are split into separate methods, with
>>>> the current request() calling both.
>>>
>>>
>>>
>>> https://github.com/kynx/zf1/commit/2e3e62c716935b712d5cea763e59c7ed1e280953
>>> I've kept (most of) my changes to subclasses - on thinking about it, if
>>> no one else has needed this functionality it's better not to clutter up the
>>> main classes.
>>>
>>>> Hopefully this will be useful to other people (particularly the E3
>>>> stream wrapper), so I'd like to keep it separate from my stream wrapper
>>>> stuff. I'll post more on that when it's ready.
>>>
>>>
>>>
>>> https://github.com/kynx/zf1/commit/39182232e3d089cf1bce63de0a3cb083c6febad1
>>> This includes the changes to Zend_Service_Rackspace_Files to support
>>> streaming requests, a Zend_Services_Rackspace_Files_Stream wrapper class,
>>> plus some stuff for handle checksums on muliple-segment uploads. On the
>>> latter, I've still got to get uploads > 5G working in the main class. I also
>>> want a checksumming class that uses proc to calculate the checksums on the
>>> fly, rather than waiting for it when the upload is complete.
>>>
>>> I'll post back when I've got some example code to show. If anyone's got
>>> any feedback it'd be greatly appreciated.
>>>
>>> Regards,
>>>
>>> Matt
>>>
>>
>