Quantcast

RFC - Mail Refactoring

classic Classic list List threaded Threaded
12 messages Options
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

RFC - Mail Refactoring

weierophinney
Administrator
Hey, all --

Dolf started work on a "Zend_Mail 2.0", but my understanding is he's too
busy currently to spearhead work on it.

I've taken the original requirements he posted, and created an
architectural outline of proposed object relations and interactions:

    http://framework.zend.com/wiki/display/ZFDEV2/RFC+-+Mail+Refactoring

I'd love to have feedback on this ASAP so we can begin work on it for
this upcoming beta2 release; the changes in architecture should make
development relatively straight-forward.

Thanks, all!

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

Artur Bodera
On Mon, Oct 31, 2011 at 7:46 PM, Matthew Weier O'Phinney
<[hidden email]>wrote:

>
>    http://framework.zend.com/wiki/display/ZFDEV2/RFC+-+Mail+Refactoring
>
>
"Transport" was more straightforward than "Mailer".
I'm personally using the word "Mailer" to describe a service that manages
mailing lists, works with queues etc.

As described, a "Mailer" is just an interface with ::send() method so it
fits
the "Transport" description much better. A Mailer might be a placeholder
for future ZF2 component for handling bulk-sending of multiple Messages.


Inline images are very common use case and belong in the Mime.
A convenience method is highly recommended, such as:
    $message->addImage(); // or
    $message->addInlineImage();

If you'd like, I can implement that for you. I've been using it for some
time
with ZF1 Mail.


Inline CSS - I'd recommend on using the "php emogrifier", which does the
best job out there:   http://www.pelagodesign.com/sidecar/emogrifier/
It's based on MIT license, we could negotiate something. I'd love for it to
be directly in the ZF2 core.

CSS parsing is more of a Message transformation, related again to Mime.
I think we could put it into Zend\Filter\MailMessageInlineCSS or create a
namespace in mail, i.e.
    Zend\Mail\Parser\InlineCSS   // or
    Zend\Mail\Transform\InlineCSS   // or

... a convenience method is always recommended:
I see 2 entry points:

    $message->addCSS(" ... ");
    $message->transformCSS();

The first one would use a CSS blob and apply it to markup when the
message body is built. The latter would try to extract <style></style> from
<head></head> and inject it into <body> markup.


A.

--
      __
     /.)\   +48 695 600 936
     \(./   [hidden email]
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

Shahar Evron-2
In reply to this post by weierophinney
Looks good - my only thought is that we should consider the shared
properties between HTTP message composition (request/response) and MIME
message composition - the structure of headers and multipart message entity
is almost the same, and a lot of the code can be shared.

One thought I noted in the Zend\Http\Client 2.0 proposal (which AFAIK is
still not fully accepted) is to abstract the HTTP message body in a way
that will allow, for example, to compose multipart message bodies from a
mix of in-memory string buffers (e.g. some form data) and an open file
stream which is read in chunks when the message is sent to avoid memory
abuse. The same concept could be useful for multipart email messages with
large attachments, but it requires that transport objects know how to
compose and send the message body when needed, or call a ->toString()
method to serialize it into a string if sending it in chunks is not
possible (e.g. not allowed by the transport layer).

Which brings me to another thought, which is that I am not sure that
composing the message into a string should be the job of the mailer object
- it works well for a self-implemented SMTP client or for using PHP's
mail() function, but may not work well for other transport adapters - for
example, if we were to implement a Web API based mail transport adapter
based on one of the commercial Web based email services such as Amazon SES.
In this case, passing a pre-composed MIME message as a string to the
transport layer will not be good, and it would be better to keep the
message as an object describing the message properties, and let the
transport layer access these properties to send the message.

Shahar.

On Mon, Oct 31, 2011 at 8:46 PM, Matthew Weier O'Phinney
<[hidden email]>wrote:

> Hey, all --
>
> Dolf started work on a "Zend_Mail 2.0", but my understanding is he's too
> busy currently to spearhead work on it.
>
> I've taken the original requirements he posted, and created an
> architectural outline of proposed object relations and interactions:
>
>    http://framework.zend.com/wiki/display/ZFDEV2/RFC+-+Mail+Refactoring
>
> I'd love to have feedback on this ASAP so we can begin work on it for
> this upcoming beta2 release; the changes in architecture should make
> development relatively straight-forward.
>
> Thanks, all!
>
> --
> Matthew Weier O'Phinney
> Project Lead            | [hidden email]
> Zend Framework          | http://framework.zend.com/
> PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
>
> --
> List: [hidden email]
> Info: http://framework.zend.com/archives
> Unsubscribe: [hidden email]
>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

weierophinney
Administrator
-- Shahar Evron <[hidden email]> wrote
(on Monday, 31 October 2011, 09:45 PM +0200):
> Looks good - my only thought is that we should consider the shared
> properties between HTTP message composition (request/response) and MIME
> message composition - the structure of headers and multipart message entity
> is almost the same, and a lot of the code can be shared.

I agree -- this is why the Header class was using "fieldValue" and
"fieldName", as I felt they were symmetric to what we have in the HTTP
component currently.

> One thought I noted in the Zend\Http\Client 2.0 proposal (which AFAIK is
> still not fully accepted) is to abstract the HTTP message body in a way
> that will allow, for example, to compose multipart message bodies from a
> mix of in-memory string buffers (e.g. some form data) and an open file
> stream which is read in chunks when the message is sent to avoid memory
> abuse. The same concept could be useful for multipart email messages with
> large attachments, but it requires that transport objects know how to
> compose and send the message body when needed, or call a ->toString()
> method to serialize it into a string if sending it in chunks is not
> possible (e.g. not allowed by the transport layer).

This is _excellent_ feedback. Once I have the basics hammered out,
perhaps we can discuss how the above could be implemented? I know it's
definitely a concern -- one of the comments on Dolf's original proposal
indicated performance/memory issues with our current implementation that
would be addressed with what you propose above.

> Which brings me to another thought, which is that I am not sure that
> composing the message into a string should be the job of the mailer object
> - it works well for a self-implemented SMTP client or for using PHP's
> mail() function, but may not work well for other transport adapters - for
> example, if we were to implement a Web API based mail transport adapter
> based on one of the commercial Web based email services such as Amazon SES.
> In this case, passing a pre-composed MIME message as a string to the
> transport layer will not be good, and it would be better to keep the
> message as an object describing the message properties, and let the
> transport layer access these properties to send the message.

I envision a few use cases, actually.

* Casting the Message to a string would return an RFC-2822 version of
  the mail -- i.e. full headers and body, serialized to a string.
* Casting the Headers to a string would return an RFC-2822
  representation of the headers.
* Casting the Attachments to a string would return an RFC-2822
  representation of the mail body.

However, the message itself is provided as an _object_ to the
Mailer/Transport, so the transport can choose to use such serialization,
or ignore it and go straight for the objects in order to send the
message.

In the case of a Sendmail implementation, the transport would serialize
the headers and content separately, and pass them to the mail() command.
In the case of an SMTP implementation, we'd serialize the entire mail to
a string and send at once. In the case of something like SES, we'd query
the Message in order to create the parameters to pass in our request to
the SES service.

So, the tl;dr: de/serialization is to provide RFC 2822 representations
of the various message parts, but transports do not need to invoke them.

> On Mon, Oct 31, 2011 at 8:46 PM, Matthew Weier O'Phinney
> <[hidden email]>wrote:
>
> > Hey, all --
> >
> > Dolf started work on a "Zend_Mail 2.0", but my understanding is he's too
> > busy currently to spearhead work on it.
> >
> > I've taken the original requirements he posted, and created an
> > architectural outline of proposed object relations and interactions:
> >
> >    http://framework.zend.com/wiki/display/ZFDEV2/RFC+-+Mail+Refactoring
> >
> > I'd love to have feedback on this ASAP so we can begin work on it for
> > this upcoming beta2 release; the changes in architecture should make
> > development relatively straight-forward.
> >
> > Thanks, all!
> >
> > --
> > Matthew Weier O'Phinney
> > Project Lead            | [hidden email]
> > Zend Framework          | http://framework.zend.com/
> > PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
> >
> > --
> > List: [hidden email]
> > Info: http://framework.zend.com/archives
> > Unsubscribe: [hidden email]
> >
> >
> >

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

weierophinney
Administrator
In reply to this post by Artur Bodera
-- Artur Bodera <[hidden email]> wrote
(on Monday, 31 October 2011, 08:39 PM +0100):

> On Mon, Oct 31, 2011 at 7:46 PM, Matthew Weier O'Phinney
> <[hidden email]>wrote:
> >    http://framework.zend.com/wiki/display/ZFDEV2/RFC+-+Mail+Refactoring
> >
> >
> "Transport" was more straightforward than "Mailer".
> I'm personally using the word "Mailer" to describe a service that manages
> mailing lists, works with queues etc.
>
> As described, a "Mailer" is just an interface with ::send() method so
> it fits the "Transport" description much better. A Mailer might be a
> placeholder for future ZF2 component for handling bulk-sending of
> multiple Messages.

Meh, it's all subjective. :)

The reason I was working with the verbiage "Mailer" is that (a) a number
of other projects use that terminology, so it would breed familiarity,
and (b) I wanted to get rid of the old terminology in order to better
illustrate the inversion in the new architecture. But "Mailer" may still
be too close to the old terminology.

I'd definitely be fine with Transport -- I just don't want folks going
and instantiating a Message object and thinking they can send it. :)

> Inline images are very common use case and belong in the Mime.
> A convenience method is highly recommended, such as:
>     $message->addImage(); // or
>     $message->addInlineImage();
>
> If you'd like, I can implement that for you. I've been using it for
> some time with ZF1 Mail.

The way ZF2 works _currently_ is that Zend\Mail\Mail extends
Zend\Mime\Message. I think that's probably still correct, though we may
need to tweak Zend\Mime a bit. If you want to implement that, I'd love
to see it -- I haven't worked with that area much.

> Inline CSS - I'd recommend on using the "php emogrifier", which does the
> best job out there:   http://www.pelagodesign.com/sidecar/emogrifier/
> It's based on MIT license, we could negotiate something. I'd love for it to
> be directly in the ZF2 core.
>
> CSS parsing is more of a Message transformation, related again to Mime.
> I think we could put it into Zend\Filter\MailMessageInlineCSS or create a
> namespace in mail, i.e.
>     Zend\Mail\Parser\InlineCSS   // or
>     Zend\Mail\Transform\InlineCSS   // or
>
> ... a convenience method is always recommended:
> I see 2 entry points:
>
>     $message->addCSS(" ... ");
>     $message->transformCSS();
>
> The first one would use a CSS blob and apply it to markup when the
> message body is built. The latter would try to extract <style></style>
> from <head></head> and inject it into <body> markup.

Why as part of a Message object? I'd argue we'd pass the message and/or
its attachments/body to a parser/transformer to do the work:

    $transform->addCss(/* ... */);
    $transform->transform($message);

This could be done in an event in order to make it opt-in, as well as to
allow pre-configuring the transformation object. This offers a better
separation of concerns and keeps the message object manageable.

Thoughts?

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

Shahar Evron-2
In reply to this post by weierophinney
On Mon, Oct 31, 2011 at 11:03 PM, Matthew Weier O'Phinney
<[hidden email]>wrote:

> -- Shahar Evron <[hidden email]> wrote
> (on Monday, 31 October 2011, 09:45 PM +0200):
> > Looks good - my only thought is that we should consider the shared
> > properties between HTTP message composition (request/response) and MIME
> > message composition - the structure of headers and multipart message
> entity
> > is almost the same, and a lot of the code can be shared.
>
> I agree -- this is why the Header class was using "fieldValue" and
> "fieldName", as I felt they were symmetric to what we have in the HTTP
> component currently.
>
> > One thought I noted in the Zend\Http\Client 2.0 proposal (which AFAIK is
> > still not fully accepted) is to abstract the HTTP message body in a way
> > that will allow, for example, to compose multipart message bodies from a
> > mix of in-memory string buffers (e.g. some form data) and an open file
> > stream which is read in chunks when the message is sent to avoid memory
> > abuse. The same concept could be useful for multipart email messages with
> > large attachments, but it requires that transport objects know how to
> > compose and send the message body when needed, or call a ->toString()
> > method to serialize it into a string if sending it in chunks is not
> > possible (e.g. not allowed by the transport layer).
>
> This is _excellent_ feedback. Once I have the basics hammered out,
> perhaps we can discuss how the above could be implemented? I know it's
> definitely a concern -- one of the comments on Dolf's original proposal
> indicated performance/memory issues with our current implementation that
> would be addressed with what you propose above.
>

I'd love that - I will try to produce an experimental implementation for
Zend\Http\Client and we can discuss how it can be reused or at least
modeled similarly.


>
> > Which brings me to another thought, which is that I am not sure that
> > composing the message into a string should be the job of the mailer
> object
> > - it works well for a self-implemented SMTP client or for using PHP's
> > mail() function, but may not work well for other transport adapters - for
> > example, if we were to implement a Web API based mail transport adapter
> > based on one of the commercial Web based email services such as Amazon
> SES.
> > In this case, passing a pre-composed MIME message as a string to the
> > transport layer will not be good, and it would be better to keep the
> > message as an object describing the message properties, and let the
> > transport layer access these properties to send the message.
>
> I envision a few use cases, actually.
>
> * Casting the Message to a string would return an RFC-2822 version of
>  the mail -- i.e. full headers and body, serialized to a string.
> * Casting the Headers to a string would return an RFC-2822
>  representation of the headers.
> * Casting the Attachments to a string would return an RFC-2822
>  representation of the mail body.
>
> However, the message itself is provided as an _object_ to the
> Mailer/Transport, so the transport can choose to use such serialization,
> or ignore it and go straight for the objects in order to send the
> message.
>

That's excellent - I think this covers it.

Shahar.


>
> In the case of a Sendmail implementation, the transport would serialize
> the headers and content separately, and pass them to the mail() command.
> In the case of an SMTP implementation, we'd serialize the entire mail to
> a string and send at once. In the case of something like SES, we'd query
> the Message in order to create the parameters to pass in our request to
> the SES service.
>
> So, the tl;dr: de/serialization is to provide RFC 2822 representations
> of the various message parts, but transports do not need to invoke them.
>
> > On Mon, Oct 31, 2011 at 8:46 PM, Matthew Weier O'Phinney
> > <[hidden email]>wrote:
> >
> > > Hey, all --
> > >
> > > Dolf started work on a "Zend_Mail 2.0", but my understanding is he's
> too
> > > busy currently to spearhead work on it.
> > >
> > > I've taken the original requirements he posted, and created an
> > > architectural outline of proposed object relations and interactions:
> > >
> > >
> http://framework.zend.com/wiki/display/ZFDEV2/RFC+-+Mail+Refactoring
> > >
> > > I'd love to have feedback on this ASAP so we can begin work on it for
> > > this upcoming beta2 release; the changes in architecture should make
> > > development relatively straight-forward.
> > >
> > > Thanks, all!
> > >
> > > --
> > > Matthew Weier O'Phinney
> > > Project Lead            | [hidden email]
> > > Zend Framework          | http://framework.zend.com/
> > > PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
> > >
> > > --
> > > List: [hidden email]
> > > Info: http://framework.zend.com/archives
> > > Unsubscribe: [hidden email]
> > >
> > >
> > >
>
> --
> Matthew Weier O'Phinney
> Project Lead            | [hidden email]
> Zend Framework          | http://framework.zend.com/
> PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
>
> --
> List: [hidden email]
> Info: http://framework.zend.com/archives
> Unsubscribe: [hidden email]
>
>
>
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

Lars Kneschke-2
In reply to this post by weierophinney
Am 31.10.2011 19:53:40, schrieb Matthew Weier O'Phinney:
Hello!
> I'd love to have feedback on this ASAP so we can begin work on it for
> this upcoming beta2 release; the changes in architecture should make
> development relatively straight-forward.



We are using the Zend_Mail classes as backend for the email client of Tine 2.0. From that perspective we see following problems:

Memory usage
When we are sending emails with attachments, we really need much memory, because the message to be sent is constructed in memory. If I remember correctly this is a problem of the smtp transport.

Different classes for SMTP and IMAP
After sending an email, its common sense, to store this email in the Sent folder. But the current implementation of the transport class does not allow to retrieve the send email. We hacked around this, but it should be possible to retrieve the message body to be sent. As best as stream.
And then you can not retrieve a message from the IMAP transport and forward this message to the SMTP transport, because they are different classes. That's very complicated. We have to parse the IMAP message and construct a class matching the SMTP transport requirements.
From my point of view it should be possible to use a common class for both use cases.

Detect supported authentication types
Currently we must define the authentication type(PLAIN, LOGIN) to use for the smtp transport. But the smtp transport can detect which authentication type is supported by the SMTP server and choose the best itself.

Encryption
It would be nice if the classes would allow to sign or encrypt messages.

We can also help solving these problems.

--
​Lars KneschkeMetaways Infosystems GmbHPickhuben 2-4, 20457 Hamburg
Tel: +49 (0)40 31 70 31 - 521Fax: +49 (0)40 31 70 31 - 921Mobile: +49 (0)175 930 4324Email: [hidden email]: www.metaways.de
Metaways Infosystems GmbH - Sitz: D-22967 TremsbüttelHandelsregister: Amtsgericht Lübeck HRB 4508 AHGeschäftsführung: Hermann Thaele, Lüder-H. Thaele
Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

weierophinney
Administrator
-- Lars Kneschke <[hidden email]> wrote
(on Tuesday, 01 November 2011, 12:23 PM +0000):

> Am 31.10.2011 19:53:40, schrieb Matthew Weier O'Phinney:
> Hello!
> > I'd love to have feedback on this ASAP so we can begin work on it for
> > this upcoming beta2 release; the changes in architecture should make
> > development relatively straight-forward.
>
>
>
> We are using the Zend_Mail classes as backend for the email client of
> Tine 2.0. From that perspective we see following problems:
>
> Memory usage
> When we are sending emails with attachments, we really need much
> memory, because the message to be sent is constructed in memory. If I
> remember correctly this is a problem of the smtp transport.

Shahar stated elsewhere in the thread some ideas for accomplishing this.

> Different classes for SMTP and IMAP
> After sending an email, its common sense, to store this email in the
> Sent folder. But the current implementation of the transport class
> does not allow to retrieve the send email. We hacked around this, but
> it should be possible to retrieve the message body to be sent. As best
> as stream.
> And then you can not retrieve a message from the IMAP transport and
> forward this message to the SMTP transport, because they are different
> classes. That's very complicated. We have to parse the IMAP message
> and construct a class matching the SMTP transport requirements.
> From my point of view it should be possible to use a common class for
> both use cases.

That's what the Message class would accomplish -- encapsulation of a
full email, allowing it to be passed back and forth between different
transports or storage classes.

> Detect supported authentication types
> Currently we must define the authentication type(PLAIN, LOGIN) to use
> for the smtp transport. But the smtp transport can detect which
> authentication type is supported by the SMTP server and choose the
> best itself.

Yes and no. Yes, you can determine the _type_ of authentication, but the
end-user still has to provide the correct type of credentials based for
the authentication type the server provides. I suppose we could allow
specifying a hash of credential type => credentials, however, to allow
auto-detection.

> Encryption
> It would be nice if the classes would allow to sign or encrypt messages.

That could be done via an event.

> We can also help solving these problems.

First steps for me would be to capture the use cases as pseudo-code,
indicating the expected behavior. Once there, and once the basic
architecture is in place, any help you can provide on implementation
will be quite welcome. :)

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

Michael Kliewe
In reply to this post by Lars Kneschke-2
Hi Lars, hi list,

we had exactly the same problems when writing a webmailer, many of them
we tried to hack around, but one of the biggest problems was the memory
usage, to fix this you would have to rewrite everything. That's why we
switched to Zeta Components, they are handling streams much better (not
perfect), a very nice way to work with big attachments, and they
directly support UIDs when working with imap storage (instead of
converting from uniqueId to number everywhere (getNumberByUniqueId),
that's annoying and not needed) and mass-commands (1 IMAP command to
move/delete/setFlag 500 mails). They are not perfect, we also have a
dozen overwritten classes and methods, but we can work with it.

Sending or reading a mail via Zend_Mail in ZF1 needs lots of memory, for
example sending a 100MB mail uses 680MB! peak memory (try it!*). To be
safe you have to give PHP a memory limit of >750MB, thats not a good
idea if you are processing many mails in parallel.

We also had this problem with storing sent mails or drafts to Sent
folder or Drafts folder, but found a workaround. But it definetly is a
problem that Zend_Mail and Zend_Mail_Message are not the same/convertable.

So when rewriting \Zend\Mail it would be perfect to learn from that and
fix all these issues. I also have some open issues regarding performance
and features, see
http://framework.zend.com/issues/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+ZF+AND+reporter+%3D+mkliewe+AND+component+in+%28Zend_Mail%2C+Zend_Mail_Storage%29

In one of those issues Nico Edtinger had the idea of "sets":
"My idea was to have a class, let's call it Zend_Mail_Storage_Set,
supporting UIDs, ranges, search, sort, ... An instance of a storage
class would have a method set(), which returns a set instance (common or
specialized for this storage). A couple of features are not IMAP
specific and it wouldn't make sense to just add a new method for one
storage.
Zend_Mail_Storage_Set would implement an iterator interface and instead
of getting just an id or UID you could also fetch the corresponding
message. I have an old mail with a couple of use cases."

If I would have more time I would help with this. The only way I can
help at the moment is writing about problems we had, and test the new
version. To name again our 3 biggest problems:
- very high memory usage when sending mails with big attachments (at the
moment up to 100MB)
- missing direct uniqueId support
- performance problems when working with many mails (see issues above)

Hope this helps
Michael

* http://pastebin.com/BdveeekP
int(686827896)
int(687603712)


Am 01.11.2011 13:23, schrieb Lars Kneschke:

> Am 31.10.2011 19:53:40, schrieb Matthew Weier O'Phinney:
> Hello!
>> I'd love to have feedback on this ASAP so we can begin work on it for
>> this upcoming beta2 release; the changes in architecture should make
>> development relatively straight-forward.
>
>
> We are using the Zend_Mail classes as backend for the email client of Tine 2.0. From that perspective we see following problems:
>
> Memory usage
> When we are sending emails with attachments, we really need much memory, because the message to be sent is constructed in memory. If I remember correctly this is a problem of the smtp transport.
>
> Different classes for SMTP and IMAP
> After sending an email, its common sense, to store this email in the Sent folder. But the current implementation of the transport class does not allow to retrieve the send email. We hacked around this, but it should be possible to retrieve the message body to be sent. As best as stream.
> And then you can not retrieve a message from the IMAP transport and forward this message to the SMTP transport, because they are different classes. That's very complicated. We have to parse the IMAP message and construct a class matching the SMTP transport requirements.
>  From my point of view it should be possible to use a common class for both use cases.
>
> Detect supported authentication types
> Currently we must define the authentication type(PLAIN, LOGIN) to use for the smtp transport. But the smtp transport can detect which authentication type is supported by the SMTP server and choose the best itself.
>
> Encryption
> It would be nice if the classes would allow to sign or encrypt messages.
>
> We can also help solving these problems.
>
> --
> ​Lars KneschkeMetaways Infosystems GmbHPickhuben 2-4, 20457 Hamburg
> Tel: +49 (0)40 31 70 31 - 521Fax: +49 (0)40 31 70 31 - 921Mobile: +49 (0)175 930 4324Email: [hidden email]: www.metaways.de
> Metaways Infosystems GmbH - Sitz: D-22967 TremsbüttelHandelsregister: Amtsgericht Lübeck HRB 4508 AHGeschäftsführung: Hermann Thaele, Lüder-H. Thaele
>


--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

weierophinney
Administrator
Going to top-post here...

Most of what you discuss has to do with the Zend\Mail\Storage
subcomponent, which I am specifically _NOT_ addressing in my proposal.

I'm focussing on the Transport subcomponent because likely > 80% of ZF
users who use Zend\Mail will only ever be _sending_ mail, and this is
the component they'll look to. For those of you building webmail clients
or doing mail processing (e.g. mail-driven events, etc.), the Storage
subcomponent becomes more key, and those of you who use it can best
identify and address the architectural issues.

One thing regarding the Transport you mention is the amount of memory it
uses -- 680MB usage for sending a 100MB email. Can you indicate what
transport you were using? Ideally we can address this using streams, but
I need to see the specific use cases (are these large attachments, or
lots of small attachments? are the contents compressed at all? etc.) in
order to address the issues.

-- Michael Kliewe <[hidden email]> wrote
(on Wednesday, 02 November 2011, 12:39 AM +0100):

> we had exactly the same problems when writing a webmailer, many of
> them we tried to hack around, but one of the biggest problems was
> the memory usage, to fix this you would have to rewrite everything.
> That's why we switched to Zeta Components, they are handling streams
> much better (not perfect), a very nice way to work with big
> attachments, and they directly support UIDs when working with imap
> storage (instead of converting from uniqueId to number everywhere
> (getNumberByUniqueId), that's annoying and not needed) and
> mass-commands (1 IMAP command to move/delete/setFlag 500 mails).
> They are not perfect, we also have a dozen overwritten classes and
> methods, but we can work with it.
>
> Sending or reading a mail via Zend_Mail in ZF1 needs lots of memory,
> for example sending a 100MB mail uses 680MB! peak memory (try it!*).
> To be safe you have to give PHP a memory limit of >750MB, thats not
> a good idea if you are processing many mails in parallel.
>
> We also had this problem with storing sent mails or drafts to Sent
> folder or Drafts folder, but found a workaround. But it definetly is
> a problem that Zend_Mail and Zend_Mail_Message are not the
> same/convertable.
>
> So when rewriting \Zend\Mail it would be perfect to learn from that
> and fix all these issues. I also have some open issues regarding
> performance and features, see
> http://framework.zend.com/issues/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+ZF+AND+reporter+%3D+mkliewe+AND+component+in+%28Zend_Mail%2C+Zend_Mail_Storage%29
>
> In one of those issues Nico Edtinger had the idea of "sets":
> "My idea was to have a class, let's call it Zend_Mail_Storage_Set,
> supporting UIDs, ranges, search, sort, ... An instance of a storage
> class would have a method set(), which returns a set instance
> (common or specialized for this storage). A couple of features are
> not IMAP specific and it wouldn't make sense to just add a new
> method for one storage.
> Zend_Mail_Storage_Set would implement an iterator interface and
> instead of getting just an id or UID you could also fetch the
> corresponding message. I have an old mail with a couple of use
> cases."
>
> If I would have more time I would help with this. The only way I can
> help at the moment is writing about problems we had, and test the
> new version. To name again our 3 biggest problems:
> - very high memory usage when sending mails with big attachments (at
> the moment up to 100MB)
> - missing direct uniqueId support
> - performance problems when working with many mails (see issues above)
>
> Hope this helps
> Michael
>
> * http://pastebin.com/BdveeekP
> int(686827896)
> int(687603712)
>
>
> Am 01.11.2011 13:23, schrieb Lars Kneschke:
> >Am 31.10.2011 19:53:40, schrieb Matthew Weier O'Phinney:
> >Hello!
> >>I'd love to have feedback on this ASAP so we can begin work on it for
> >>this upcoming beta2 release; the changes in architecture should make
> >>development relatively straight-forward.
> >
> >
> >We are using the Zend_Mail classes as backend for the email client of Tine 2.0. From that perspective we see following problems:
> >
> >Memory usage
> >When we are sending emails with attachments, we really need much memory, because the message to be sent is constructed in memory. If I remember correctly this is a problem of the smtp transport.
> >
> >Different classes for SMTP and IMAP
> >After sending an email, its common sense, to store this email in the Sent folder. But the current implementation of the transport class does not allow to retrieve the send email. We hacked around this, but it should be possible to retrieve the message body to be sent. As best as stream.
> >And then you can not retrieve a message from the IMAP transport and forward this message to the SMTP transport, because they are different classes. That's very complicated. We have to parse the IMAP message and construct a class matching the SMTP transport requirements.
> > From my point of view it should be possible to use a common class for both use cases.
> >
> >Detect supported authentication types
> >Currently we must define the authentication type(PLAIN, LOGIN) to use for the smtp transport. But the smtp transport can detect which authentication type is supported by the SMTP server and choose the best itself.
> >
> >Encryption
> >It would be nice if the classes would allow to sign or encrypt messages.
> >
> >We can also help solving these problems.
> >
> >--
> >​Lars KneschkeMetaways Infosystems GmbHPickhuben 2-4, 20457 Hamburg
> >Tel: +49 (0)40 31 70 31 - 521Fax: +49 (0)40 31 70 31 - 921Mobile: +49 (0)175 930 4324Email: [hidden email]: www.metaways.de
> >Metaways Infosystems GmbH - Sitz: D-22967 TremsbüttelHandelsregister: Amtsgericht Lübeck HRB 4508 AHGeschäftsführung: Hermann Thaele, Lüder-H. Thaele
> >
>
>
> --
> List: [hidden email]
> Info: http://framework.zend.com/archives
> Unsubscribe: [hidden email]
>
>

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

Michael Kliewe
Hi Matthew,

you are right, I'm mixing storage with mail sending, but in some cases they are connected (putting a copy to Sent/Drafts folder, or reading a mail from Drafts folder and send it).
I provided a link to pastebin in the email below with the script I used to test that. I used one big attachment and SMTP transport. I am providing a stream to the createAttachment method, but internally it is not used, the whole attachment content is put into variables, copied/encoded and put into the whole generated mail string and so on, that's why it grows so heavily.

http://pastebin.com/BdveeekP

Michael

On Nov 2, 2011, at 2:50 PM, Matthew Weier O'Phinney wrote:

> Going to top-post here...
>
> Most of what you discuss has to do with the Zend\Mail\Storage
> subcomponent, which I am specifically _NOT_ addressing in my proposal.
>
> I'm focussing on the Transport subcomponent because likely > 80% of ZF
> users who use Zend\Mail will only ever be _sending_ mail, and this is
> the component they'll look to. For those of you building webmail clients
> or doing mail processing (e.g. mail-driven events, etc.), the Storage
> subcomponent becomes more key, and those of you who use it can best
> identify and address the architectural issues.
>
> One thing regarding the Transport you mention is the amount of memory it
> uses -- 680MB usage for sending a 100MB email. Can you indicate what
> transport you were using? Ideally we can address this using streams, but
> I need to see the specific use cases (are these large attachments, or
> lots of small attachments? are the contents compressed at all? etc.) in
> order to address the issues.
>
> -- Michael Kliewe <[hidden email]> wrote
> (on Wednesday, 02 November 2011, 12:39 AM +0100):
>> we had exactly the same problems when writing a webmailer, many of
>> them we tried to hack around, but one of the biggest problems was
>> the memory usage, to fix this you would have to rewrite everything.
>> That's why we switched to Zeta Components, they are handling streams
>> much better (not perfect), a very nice way to work with big
>> attachments, and they directly support UIDs when working with imap
>> storage (instead of converting from uniqueId to number everywhere
>> (getNumberByUniqueId), that's annoying and not needed) and
>> mass-commands (1 IMAP command to move/delete/setFlag 500 mails).
>> They are not perfect, we also have a dozen overwritten classes and
>> methods, but we can work with it.
>>
>> Sending or reading a mail via Zend_Mail in ZF1 needs lots of memory,
>> for example sending a 100MB mail uses 680MB! peak memory (try it!*).
>> To be safe you have to give PHP a memory limit of >750MB, thats not
>> a good idea if you are processing many mails in parallel.
>>
>> We also had this problem with storing sent mails or drafts to Sent
>> folder or Drafts folder, but found a workaround. But it definetly is
>> a problem that Zend_Mail and Zend_Mail_Message are not the
>> same/convertable.
>>
>> So when rewriting \Zend\Mail it would be perfect to learn from that
>> and fix all these issues. I also have some open issues regarding
>> performance and features, see
>> http://framework.zend.com/issues/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+ZF+AND+reporter+%3D+mkliewe+AND+component+in+%28Zend_Mail%2C+Zend_Mail_Storage%29
>>
>> In one of those issues Nico Edtinger had the idea of "sets":
>> "My idea was to have a class, let's call it Zend_Mail_Storage_Set,
>> supporting UIDs, ranges, search, sort, ... An instance of a storage
>> class would have a method set(), which returns a set instance
>> (common or specialized for this storage). A couple of features are
>> not IMAP specific and it wouldn't make sense to just add a new
>> method for one storage.
>> Zend_Mail_Storage_Set would implement an iterator interface and
>> instead of getting just an id or UID you could also fetch the
>> corresponding message. I have an old mail with a couple of use
>> cases."
>>
>> If I would have more time I would help with this. The only way I can
>> help at the moment is writing about problems we had, and test the
>> new version. To name again our 3 biggest problems:
>> - very high memory usage when sending mails with big attachments (at
>> the moment up to 100MB)
>> - missing direct uniqueId support
>> - performance problems when working with many mails (see issues above)
>>
>> Hope this helps
>> Michael
>>
>> * http://pastebin.com/BdveeekP
>> int(686827896)
>> int(687603712)
>>
>>
>> Am 01.11.2011 13:23, schrieb Lars Kneschke:
>>> Am 31.10.2011 19:53:40, schrieb Matthew Weier O'Phinney:
>>> Hello!
>>>> I'd love to have feedback on this ASAP so we can begin work on it for
>>>> this upcoming beta2 release; the changes in architecture should make
>>>> development relatively straight-forward.
>>>
>>>
>>> We are using the Zend_Mail classes as backend for the email client of Tine 2.0. From that perspective we see following problems:
>>>
>>> Memory usage
>>> When we are sending emails with attachments, we really need much memory, because the message to be sent is constructed in memory. If I remember correctly this is a problem of the smtp transport.
>>>
>>> Different classes for SMTP and IMAP
>>> After sending an email, its common sense, to store this email in the Sent folder. But the current implementation of the transport class does not allow to retrieve the send email. We hacked around this, but it should be possible to retrieve the message body to be sent. As best as stream.
>>> And then you can not retrieve a message from the IMAP transport and forward this message to the SMTP transport, because they are different classes. That's very complicated. We have to parse the IMAP message and construct a class matching the SMTP transport requirements.
>>> From my point of view it should be possible to use a common class for both use cases.
>>>
>>> Detect supported authentication types
>>> Currently we must define the authentication type(PLAIN, LOGIN) to use for the smtp transport. But the smtp transport can detect which authentication type is supported by the SMTP server and choose the best itself.
>>>
>>> Encryption
>>> It would be nice if the classes would allow to sign or encrypt messages.
>>>
>>> We can also help solving these problems.
>>>
>>> --
>>> ​Lars KneschkeMetaways Infosystems GmbHPickhuben 2-4, 20457 Hamburg
>>> Tel: +49 (0)40 31 70 31 - 521Fax: +49 (0)40 31 70 31 - 921Mobile: +49 (0)175 930 4324Email: [hidden email]: www.metaways.de
>>> Metaways Infosystems GmbH - Sitz: D-22967 TremsbüttelHandelsregister: Amtsgericht Lübeck HRB 4508 AHGeschäftsführung: Hermann Thaele, Lüder-H. Thaele
>>>
>>
>>
>> --
>> List: [hidden email]
>> Info: http://framework.zend.com/archives
>> Unsubscribe: [hidden email]
>>
>>
>
> --
> Matthew Weier O'Phinney
> Project Lead            | [hidden email]
> Zend Framework          | http://framework.zend.com/
> PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
>
> --
> List: [hidden email]
> Info: http://framework.zend.com/archives
> Unsubscribe: [hidden email]
>
>


--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Reply | Threaded
Open this post in threaded view
|  
Report Content as Inappropriate

Re: RFC - Mail Refactoring

weierophinney
Administrator
-- Michael Kliewe <[hidden email]> wrote
(on Wednesday, 02 November 2011, 03:20 PM +0100):

> you are right, I'm mixing storage with mail sending, but in some cases
> they are connected (putting a copy to Sent/Drafts folder, or reading a
> mail from Drafts folder and send it).
> I provided a link to pastebin in the email below with the script I
> used to test that. I used one big attachment and SMTP transport. I am
> providing a stream to the createAttachment method, but internally it
> is not used, the whole attachment content is put into variables,
> copied/encoded and put into the whole generated mail string and so on,
> that's why it grows so heavily.
>
> http://pastebin.com/BdveeekP

Awesome - that's a great use case we can use in the testbed. :)


> On Nov 2, 2011, at 2:50 PM, Matthew Weier O'Phinney wrote:
>
> > Going to top-post here...
> >
> > Most of what you discuss has to do with the Zend\Mail\Storage
> > subcomponent, which I am specifically _NOT_ addressing in my proposal.
> >
> > I'm focussing on the Transport subcomponent because likely > 80% of ZF
> > users who use Zend\Mail will only ever be _sending_ mail, and this is
> > the component they'll look to. For those of you building webmail clients
> > or doing mail processing (e.g. mail-driven events, etc.), the Storage
> > subcomponent becomes more key, and those of you who use it can best
> > identify and address the architectural issues.
> >
> > One thing regarding the Transport you mention is the amount of memory it
> > uses -- 680MB usage for sending a 100MB email. Can you indicate what
> > transport you were using? Ideally we can address this using streams, but
> > I need to see the specific use cases (are these large attachments, or
> > lots of small attachments? are the contents compressed at all? etc.) in
> > order to address the issues.
> >
> > -- Michael Kliewe <[hidden email]> wrote
> > (on Wednesday, 02 November 2011, 12:39 AM +0100):
> >> we had exactly the same problems when writing a webmailer, many of
> >> them we tried to hack around, but one of the biggest problems was
> >> the memory usage, to fix this you would have to rewrite everything.
> >> That's why we switched to Zeta Components, they are handling streams
> >> much better (not perfect), a very nice way to work with big
> >> attachments, and they directly support UIDs when working with imap
> >> storage (instead of converting from uniqueId to number everywhere
> >> (getNumberByUniqueId), that's annoying and not needed) and
> >> mass-commands (1 IMAP command to move/delete/setFlag 500 mails).
> >> They are not perfect, we also have a dozen overwritten classes and
> >> methods, but we can work with it.
> >>
> >> Sending or reading a mail via Zend_Mail in ZF1 needs lots of memory,
> >> for example sending a 100MB mail uses 680MB! peak memory (try it!*).
> >> To be safe you have to give PHP a memory limit of >750MB, thats not
> >> a good idea if you are processing many mails in parallel.
> >>
> >> We also had this problem with storing sent mails or drafts to Sent
> >> folder or Drafts folder, but found a workaround. But it definetly is
> >> a problem that Zend_Mail and Zend_Mail_Message are not the
> >> same/convertable.
> >>
> >> So when rewriting \Zend\Mail it would be perfect to learn from that
> >> and fix all these issues. I also have some open issues regarding
> >> performance and features, see
> >> http://framework.zend.com/issues/secure/IssueNavigator.jspa?reset=true&jqlQuery=project+%3D+ZF+AND+reporter+%3D+mkliewe+AND+component+in+%28Zend_Mail%2C+Zend_Mail_Storage%29
> >>
> >> In one of those issues Nico Edtinger had the idea of "sets":
> >> "My idea was to have a class, let's call it Zend_Mail_Storage_Set,
> >> supporting UIDs, ranges, search, sort, ... An instance of a storage
> >> class would have a method set(), which returns a set instance
> >> (common or specialized for this storage). A couple of features are
> >> not IMAP specific and it wouldn't make sense to just add a new
> >> method for one storage.
> >> Zend_Mail_Storage_Set would implement an iterator interface and
> >> instead of getting just an id or UID you could also fetch the
> >> corresponding message. I have an old mail with a couple of use
> >> cases."
> >>
> >> If I would have more time I would help with this. The only way I can
> >> help at the moment is writing about problems we had, and test the
> >> new version. To name again our 3 biggest problems:
> >> - very high memory usage when sending mails with big attachments (at
> >> the moment up to 100MB)
> >> - missing direct uniqueId support
> >> - performance problems when working with many mails (see issues above)
> >>
> >> Hope this helps
> >> Michael
> >>
> >> * http://pastebin.com/BdveeekP
> >> int(686827896)
> >> int(687603712)
> >>
> >>
> >> Am 01.11.2011 13:23, schrieb Lars Kneschke:
> >>> Am 31.10.2011 19:53:40, schrieb Matthew Weier O'Phinney:
> >>> Hello!
> >>>> I'd love to have feedback on this ASAP so we can begin work on it for
> >>>> this upcoming beta2 release; the changes in architecture should make
> >>>> development relatively straight-forward.
> >>>
> >>>
> >>> We are using the Zend_Mail classes as backend for the email client of Tine 2.0. From that perspective we see following problems:
> >>>
> >>> Memory usage
> >>> When we are sending emails with attachments, we really need much memory, because the message to be sent is constructed in memory. If I remember correctly this is a problem of the smtp transport.
> >>>
> >>> Different classes for SMTP and IMAP
> >>> After sending an email, its common sense, to store this email in the Sent folder. But the current implementation of the transport class does not allow to retrieve the send email. We hacked around this, but it should be possible to retrieve the message body to be sent. As best as stream.
> >>> And then you can not retrieve a message from the IMAP transport and forward this message to the SMTP transport, because they are different classes. That's very complicated. We have to parse the IMAP message and construct a class matching the SMTP transport requirements.
> >>> From my point of view it should be possible to use a common class for both use cases.
> >>>
> >>> Detect supported authentication types
> >>> Currently we must define the authentication type(PLAIN, LOGIN) to use for the smtp transport. But the smtp transport can detect which authentication type is supported by the SMTP server and choose the best itself.
> >>>
> >>> Encryption
> >>> It would be nice if the classes would allow to sign or encrypt messages.
> >>>
> >>> We can also help solving these problems.
> >>>
> >>> --
> >>> ​Lars KneschkeMetaways Infosystems GmbHPickhuben 2-4, 20457 Hamburg
> >>> Tel: +49 (0)40 31 70 31 - 521Fax: +49 (0)40 31 70 31 - 921Mobile: +49 (0)175 930 4324Email: [hidden email]: www.metaways.de
> >>> Metaways Infosystems GmbH - Sitz: D-22967 TremsbüttelHandelsregister: Amtsgericht Lübeck HRB 4508 AHGeschäftsführung: Hermann Thaele, Lüder-H. Thaele
> >>>
> >>
> >>
> >> --
> >> List: [hidden email]
> >> Info: http://framework.zend.com/archives
> >> Unsubscribe: [hidden email]
> >>
> >>
> >
> > --
> > Matthew Weier O'Phinney
> > Project Lead            | [hidden email]
> > Zend Framework          | http://framework.zend.com/
> > PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc
> >
> > --
> > List: [hidden email]
> > Info: http://framework.zend.com/archives
> > Unsubscribe: [hidden email]
> >
> >
>

--
Matthew Weier O'Phinney
Project Lead            | [hidden email]
Zend Framework          | http://framework.zend.com/
PGP key: http://framework.zend.com/zf-matthew-pgp-key.asc

--
List: [hidden email]
Info: http://framework.zend.com/archives
Unsubscribe: [hidden email]


Loading...